<table><caption>Table 3.1: List of the most relevant contributions to the Few-Shot Object Detection field. These works are grouped according to the general approach employed to tackle FSOD and sorted by their year of publication. Green rows signify that the methods were applied to aerial images and § indicates that these images are SAR images. † signals that it was applied to generalized FSOD while ‡ means that it was developed in an incremental setting.</caption><thead><tr><th>Approach</th><th>Abbreviation</th><th>Venue</th><th>Date</th><th>Detection Framework</th><th>Multiscale</th><th>Datasets</th><th>Aerial / Natural Images</th></tr></thead><tbody><tr><td rowspan="16">Attention</td><td>FRW [220]</td><td>ICCV</td><td>2019</td><td>YOLO</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>OSOD-CACE [221]</td><td>NEURIPS</td><td>2019</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>Meta R-CNN [222]</td><td>ICCV</td><td>2019</td><td>Faster RCNN</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>FSOD-RSI [223]</td><td>TGRS</td><td>2020</td><td>YOLO</td><td>Yes</td><td>DIOR / NWPU VHR</td><td>Aerial</td></tr><tr><td>ARPN [224]</td><td>CVPR</td><td>2020</td><td>Faster RCNN</td><td>Yes</td><td>COCO</td><td>Natural</td></tr><tr><td>VEOW [225]</td><td>ECCV</td><td>2020</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>KT [226]</td><td>SMC</td><td>2020</td><td>Faster RCNN</td><td>Yes</td><td>Pascal</td><td>Natural</td></tr><tr><td>OSOD-WFT [227]</td><td>Preprint</td><td>2020</td><td>FCOS</td><td>Yes</td><td>Pascal / COCO / ImageNet Loc</td><td>Natural</td></tr><tr><td>ONCE [228]†</td><td>CVPR</td><td>2020</td><td>Center Net</td><td>No</td><td>Pascal / COCO / Deepfashion</td><td>Natural</td></tr><tr><td>WSAAN [229]</td><td>TAEORS</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>RSOD / NWPU VHR</td><td>Aerial</td></tr><tr><td>FSOD-FPDI [230]</td><td>MDPI</td><td>2021</td><td>FCOS</td><td>Yes</td><td>DOTA / NWPU VHR</td><td>Aerial</td></tr><tr><td>Meta-FRCNN [231]</td><td>AAAI</td><td>2022</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>Meta-DETR [232]</td><td>TPAMI</td><td>2021</td><td>DETR</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>DRL [233]</td><td>Preprint</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>DANA [234]</td><td>TM</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>SP [235]</td><td>Access</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>JCACR [236]</td><td>ICIP</td><td>2021</td><td>YOLO</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>TI-FSOD [237]</td><td>CVPR</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>SAM [238]</td><td>MDPI</td><td>2021</td><td>Faster RCNN</td><td>No</td><td>NWPU VHR-10 / DIOR</td><td>Aerial</td></tr><tr><td>FSOD-FCT [239]</td><td>CVPR</td><td>2022</td><td>Faster RCNN</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>SAR-DRM [240]</td><td>TGRS</td><td>2022</td><td>Faster RCNN</td><td>No</td><td>FUSAR-GEN</td><td>Aerial §</td></tr><tr><td>FSOD-PSI [241]</td><td>JDT</td><td>2022</td><td>YOLO</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>SAFT [242]</td><td>CVPR</td><td>2022</td><td>FCOS</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>APSP [243]</td><td>WACV</td><td>2022</td><td>Faster RCNN</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>KFSOD [244]</td><td>CVPR</td><td>2022</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>FSODS [245]</td><td>TGRS</td><td>2022</td><td>YOLO</td><td>Yes</td><td>SMCDD-FS</td><td>Aerial §</td></tr><tr><td>TIN-FSOD [246]</td><td>Arxiv</td><td>2023</td><td>Faster RCNN</td><td>Yes</td><td>NWPU VHR / DIOR / HRRSD</td><td>Aerial</td></tr><tr><td>FSOD-ICF [247]</td><td>WACV</td><td>2023</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td rowspan="3">Attention / Metric Learning</td><td>PNPDet [248]</td><td>WACV</td><td>2021</td><td>Center Net</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>UPE [249]</td><td>ICCV</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>GenDet [250]</td><td>NNLS</td><td>2021</td><td>FCOS</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td rowspan="10">Metric learning</td><td>RepMet [251]</td><td>CVPR</td><td>2018</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / ImageNet Loc</td><td>Natural</td></tr><tr><td>RN-FSOD [252]</td><td>NEURIPS</td><td>2020</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / ImageNet Loc</td><td>Natural</td></tr><tr><td>MDODD [253]†</td><td>ICCV</td><td>2021</td><td>Faster RCNN</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>FSCE [254]</td><td>CVPR</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>GD-FSOD [255]</td><td>NEURIPS</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>LSTD [256]</td><td>AAAI</td><td>2018</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO / ImageNet Loc</td><td>Natural</td></tr><tr><td>MSPSR [257]</td><td>ECCV</td><td>2020</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>TFA [258]</td><td>ICML</td><td>2020</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO / LVIS</td><td>Natural</td></tr><tr><td>WOFG [259]†</td><td>CVPR</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>Hallu-FSOD [260]</td><td>CVPR</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td rowspan="10">Fine-tuning Strategy</td><td>DHP [261]</td><td>ICCVW</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>iSAID / NWPU VHR</td><td>Aerial</td></tr><tr><td>LVC [262]</td><td>CVPR</td><td>2021</td><td>Faster RCNN</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>FSCN [263]</td><td>CVPR</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>FADI [264]</td><td>NEURIPS</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>DeFRCN [265]</td><td>ICCV</td><td>2021</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>SIMPL [266]</td><td>TAEORS</td><td>2022</td><td>YOLO</td><td>No</td><td>xView (plane only)</td><td>Aerial</td></tr><tr><td>DETReg [267]</td><td>CVPR</td><td>2022</td><td>Deformable DETR</td><td>Yes</td><td>COCO</td><td>Natural</td></tr><tr><td>CFA [268]†</td><td>CVPRW</td><td>2022</td><td>Faster RCNN</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>CIR [269]</td><td>TGRS</td><td>2022</td><td>Faster RCNN</td><td>Yes</td><td>NWPU VHR-10 / DIOR</td><td>Aerial</td></tr><tr><td>NIMPE [270]</td><td>ICASSP</td><td>2022</td><td>Faster RCNN</td><td>Yes</td><td>COCO</td><td>Natural</td></tr><tr><td>HDA [271]</td><td>IROS</td><td>2022</td><td>Faster RCNN</td><td>Yes</td><td>COCO</td><td>Natural</td></tr><tr><td>MDB [272]</td><td>LNCS</td><td>2022</td><td>Faster RCNN</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>DCB [273]†</td><td>NEURIPS</td><td>2022</td><td>Faster RCNN</td><td>Yes</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td rowspan="2">Meta-Learning</td><td>MetaDet [276]</td><td>ICCV</td><td>2019</td><td>Faster RCNN</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td>Sylph [277]†</td><td>CVPR</td><td>2022</td><td>Faster RCNN</td><td>No</td><td>COCO / LVIS</td><td>Natural</td></tr><tr><td rowspan="2">Zero-shot Object Detection</td><td>TL-ZSOD [278]</td><td>ICCV</td><td>2019</td><td>RetinaNet</td><td>Yes</td><td>COCO</td><td>Natural</td></tr><tr><td>ML-CMP [279]</td><td>Preprint</td><td>2022</td><td>Faster RCNN</td><td>No</td><td>Pascal / COCO</td><td>Natural</td></tr><tr><td rowspan="6">Cross-domain</td><td>OA-FSUI2IT [280]</td><td>AAAI</td><td>2022</td><td>Faster RCNN</td><td>Yes</td><td>Multiple datasets</td><td>Natural</td></tr><tr><td>Acro FOD [281]</td><td>ECCV</td><td>2022</td><td>YOLO</td><td>Yes</td><td>Multiple datasets</td><td>Natural</td></tr><tr><td>CD-CutMix [282]</td><td>ACCV</td><td>2022</td><td>Faster RCNN</td><td>No</td><td>Multiple datasets</td><td>Natural</td></tr><tr><td>CD-FSOD [283]</td><td>Preprint</td><td>2022</td><td>Faster RCNN</td><td>Yes</td><td>Multiple datasets</td><td>Aerial</td></tr><tr><td>CD-MDB [284]</td><td>ECCV</td><td>2022</td><td>Faster RCNN</td><td>Yes</td><td>Multiple datasets</td><td>Aerial</td></tr></tbody></table>
on novel classes with the few available annotations. This is leveraged by Low Shot Transfer Detector (LSTD) [256], a pioneer work on FSOD. It first trains a Faster R-CNN on a base dataset and fine-tunes it on a support set containing only some examples of the novel classes. Regularization losses are introduced to prevent overfitting. Before fine-tuning, the last layer of the classifier branch is replaced with a randomly initialized layer with the right number of outputs (i.e., the number of novel classes |Cnovel|). Closely related, [258] leverages the same idea without any additional loss. Instead, they propose a Two-stage Fine-tuning Approach (TFA), which freezes most of the network after base training. TFA is then extended by Constraint-based Fine-tuning Approach (CFA) [268] which leverages a technique borrowed from Continual Learning: Average Gradient Episodic Memory. It applies orthogonality constraints to the gradient during fine-tuning to prevent forgetting base knowledge. This mostly helps for the generalized FSOD setting, but it is also beneficial for regular FSOD. Another extension on top of the basic fine-tuning approaches is to add a refinement step to filter the bounding boxes predicted by the fine-tuned network. For instance, [263] proposes a Few-Shot Correction Network (FSCN) whose goal is to assist the detector classification branch. It is trained directly on the false positive of the detector to specifically target challenging situations. Similarly, [262] leverages a kNN classifier to “verify” the predicted labels and lightweight bounding regressors to “correct” the predicted localizations. Multi-scale Positive Sample Refinement (MSPSR) [257] also proposes a proposal refinement strategy by leveraging a multiscale refinement branch. It provides a better balance between positive and negative samples and makes both base training and fine-tuning more efficient.

Another line of work addresses the FSOD problem through an augmentation perspective. It circumvents the low few-annotated examples and the overfitting risk by enriching the support set with more or less elaborated augmentations. An easy and effective solution is to crop and paste novel instances directly inside base images [274]. During fine-tuning, images from the base dataset and support examples are randomly sampled. The support examples are cropped and pasted into the base images. This significantly boosts the fine-tuning procedure and improves FSOD performance. Similarly, Synthetic object IMPLantation (SIMPL) [266] leverages 3D models for each novel class to generate high-quality augmented images. SIMPL completely blends the augmented object inside the image, whereas [274] pastes some background around the novel object as well. SIMPL leverages external information about the classes and requires access to 3D models of the classes which is not always possible. However, this opens an opportunity for addressing the even more challenging zero-shot setting. Pushing even further, [260] proposes a generative model to enrich the support set and improve detection quality. The *hallucinator* model is trained jointly with the detector in an EM-like procedure. First, the hallucinator is trained with the detector classification loss (the detector is kept frozen). Then, the detector is trained while the hallucinator provides more support examples (with the hallucinator now frozen).

Finally, some other works leverage the fine-tuning strategy with other tricks. Novel Instances Mining with Pseudo-Margin Evaluation (NIMPE) [270] build a mining network to extract pseudo-labels
from the base dataset. Reference [272] fine-tunes both the classifier and the regressor of Faster R-CNN with an additional distillation loss based on pseudo-labels. Pseudo-labels are computed in a metric learning fashion between the query feature and the prototype features stored in a memory bank. Few-shot object detection via Association and DIscrimination (FADI) [264] splits the fine-tuning step into association and discrimination steps. During the association step, the network is fine-tuned to map novel classes onto base classes. It leverages the well-structured base class representation space learned during base training and separates novel classes. Then, the discrimination step disentangles base and novel class representations with a dedicated margin loss.

Considering FSOD as a hierarchical refinement [271] is also a viable option as it breaks down base classes into novel classes. While this setup is certainly relevant for many applications, it differs from the commonly adopted FSOD setting.

### 3.2.2 Metric-learning based methods

Metric-learning-based methods are extensively employed for few-shot classification. Metric learning is designed especially for classification and cannot handle bounding box regression. Thus, it cannot be directly applied to object detection. However, several attempts were made to tackle FSOD with metric-learning techniques, mostly replacing the classification branch of the model with prototypical networks or closely related variants and keeping the regression branch unchanged. Of course, even the classification adaptation is not straightforward as object detection includes a special background class that should be processed with care. Among these attempts, RepMet [251] learns class representative vectors to classify Regions of Interest (RoI) in Faster R-CNN according to their distance to the closest class prototype. Class vectors are initialized with support image representations and then fine-tuned via backpropagation. The fine-tuning is based on a cross-entropy loss and a margin metric loss which favors tight clusters in the embedding space. The background class probability is computed as the complementary probability of the most probable class. Closely related, Plug-and-Play Detectors (PNPDet) [248] learns prototype vectors as well as scale factors. In addition, they replace the Euclidean distance from RepMet with a Cosine similarity measure. Similarly, FSCE [254] adds a contrastive head on top of a pre-trained detector during fine-tuning. This head outputs embedding for each RoI. A contrastive loss is optimized to bring closer the representations of same-class RoI and repel RoI with no objects. Likewise, [253] leverages prototypes as well but deals with the background class separately with a learnable binary classifier.

Plenty of other works leverage class prototypes for the classification part of the detector. However, various tricks are proposed in the literature to improve the quality and use of the prototypes. For instance, Universal Prototype Enhancement (UPE) [249] refines prototypes with affine transformation to convert image-level representations into object-level prototypes much more adapted to the detection task. Also, it does not leverage the prototypes directly as a classifier but rather uses them to adapt query features before classification and regression. Similarly, GenDet (Generate Detectors from Few Shots) [250] combines the technique from RepMet and UPE, i.e., learnable prototypes to
adapt the query features. Negative prototypes can also be learned to better deal with the back-
ground class [252]. Finally, some contributions manage two sets of prototypes, arguing that one
set is not optimal for adapting features for both the classification and the regression. Decoupled
Metric Network [285] introduces a decoupled representation transformation to adapt class proto-
types for either classification or regression. Likewise, [255] splits the representations using Singular
Value Decomposition. Eigenvectors corresponding to the largest singular values represent the main
source of variance. The authors argue that this accounts for the general adaptation between base
and novel classes. They are leveraged for adapting query features both for regression and classifica-
tion. Other eigenvectors only represent the inter-class variance and therefore, are only used in the
classification branch. The methods described in this paragraph are slightly different from the ones
at the beginning of this section. They all use their representation vectors to update query features
before a learnable classification and regression module, instead of using them for direct classifica-
tion (e.g., distance to the closest prototype). They highly resemble some attention-based methods
that will be presented in the next section.

**3.2.3 Attention-based methods**

As we briefly broached at the end of the previous section, a common technique for FSOD is to adapt
the features from the query image based on the support images. This can be understood as an at-
tention mechanism between the query and support features as it highlights locations in the query
feature map that are similar to the support images. Another way to see this is to think of the at-
tention mechanism as an adaptive filtering layer. It filters the query features map according to the
support features. Highlighted locations in the query map show features similar to the support im-
ages. Following the attention module, the regression and classification are performed independently
per class, often using a shared, class-agnostic detection head (see Fig. 3.2). The query-support com-
bination module takes the query feature maps and the features from all novel classes as input and
outputs class-specific query feature maps. This will be explained in more detail in Chap. 6 which
presents a general framework to subsume existing attention mechanisms for FSOD. To summarize,
what we call here attention-based FSOD methods are techniques that leverage support information
to adapt query features before the classification and regression branches. Following this definition
the methods presented at the end of Sec. 3.2.2 can be interpreted as attention methods. However,
they are presented from a metric learning perspective which is why they are not discussed in the
current section (they will be classified as "Attention/Metric Learning" methods in Tab. 3.1).

A seminal work in this field introduces Feature ReWeighting (FRW) [220], which trains a reweight-
ing module along with a YOLO detector. The reweighting module produces class-specific feature
vectors with a Global Pooling layer (GP) applied on the support feature maps. These are then
channel-wise multiplied by the query features extracted by the backbone. Hence, class-specific
query features are generated, and the detection head computes predictions for each class separately.
This technique has been widely re-used in the following literature, with other detection frameworks:
Faster R-CNN [286, 287, 229, 225, 226, 221], CenterNet [228, 248] or FCOS [227]. The class reweight-
<figure><img src="image_51.png" /><figcaption>Figure 3.2: Attention-based FSOD principle</figcaption></figure>

ing vectors can be enriched by several tricks to improve feature filtering. A few works [226, 229, 233] employ Graph Convolutional Networks (GCNs) to combine and refine the reweighting vectors before the combination modules. [230] finds optimal vectors through iterative optimization. Others leverage multi-scale features to enrich class representation [223, 230].

This channel-wise multiplication between query and support features is a simple form of attention. It can be thought of as an adaptive convolution layer, whose weights depend on the support features. Incidentally, it is often interpreted and implemented as such by existing works, approaching the meta-learning paradigm. However, more complex attention mechanisms have been leveraged in the literature. The incentive behind this improvement was the loss of spatial support information and background feature contamination with the GP layer. First, [235] proposes a self-attention module to better highlight the support object features and prevent background contamination. Very similar, Dual AwareNess Attention (DANA) [234] introduces a background attenuation block for the same reason. However, DANA also leverages an alignment mechanism to combine query and support features without losing spatial information. This alignment module is quite close to the visual transformers' attention. It encodes the features from the query images as *queries* and the features from a support image as *keys* and *values*. *Queries* here refer to the query-key-value (QKV) formulation of the transformers, the correspondence with the query features is fortuitous. *Queries* and *keys* are combined to form an attention map, which represents the similarity between the query and support image patches. Then, the dot product between the attention map and the values produces the aligned support features. It can be understood as an alignment as it re-organizes spatially the support features to match the spatial dimension of the query map, according to the similarity between query and support. The underlying idea is that the same class objects in the query and support images will likely have different aspects or poses. Therefore, a direct comparison between the feature maps is often irrelevant. The alignment procedure moves support features to similar locations in the query map. This is illustrated in Fig. 3.3, but more details will be given in Chap. 6.
<figure><img src="image_52.png" /><figcaption>Figure 3.3: Spatial alignment between query and support feature maps. Similarity matrix is computed as an outer product between the feature maps. For sake of clarity, maps are reshaped as 2-D matrix where the first dimension controls the spatial position in the map: $n_q$ positions for the query and $n_s$ for the support. $d$ is the number of channels. Similar colors mean that features are similar.</figcaption></figure>

Hence, this alignment mechanism combines query and support features, highlighting their similarity without losing spatial information. The same technique is leveraged by several works [286, 231, 239, 242] with slight variations. Similarly, [243] uses QKV attention with globally pooled support features. Instead, the authors propose to compute attention between the query images and all support images for a class at once. In other works, the attention is generally aggregated per class. Following the same idea, Meta-DETR [232] computes attention between a query image and all support images at once. However, the authors do this for all classes at the same time and replace the binary classification with a multi-class classification layer (unlike most methods discussed above). To achieve this, a task encoding module adapts the features for a specific task (i.e., to the classes of interest) before the classification head.

Of course, these are not the only attention mechanisms existing in the FSOD literature. Some works derive other kinds of attention achieving competitive results. Kernelized FSOD (KFSOD) [244] proposes elaborated kernel functions to combine query and support features in various ways, which can be interpreted as attention. Differently, [287] trains three distinct branches that combine query and support in different fashions, globally, locally, and patch-to-patch. Dynamic Relevance Learning (DRL) [233] proposes a simpler way to combine query and support features by simple point-wise operations (concatenation, multiplication, and subtraction) after global pooling.

In addition to these attention mechanisms, some works also propose additional loss functions to improve the quality of the extracted features (query and support). As an example, Transformation Invariant FSOD (TI-FSOD) [237] leverages two losses to enforce robust feature extraction. These losses are implemented as a distance between original and augmented query or support features. The same principle is also proposed for application on remote sensing images by [246]. Another technique is proposed by [247], which computes a regularization loss between two randomly sampled subsets of an RoI feature. This regularization enforces consistency and robustness in the feature space making the detection easier. Likewise, [236] derives a reconstruction loss function by com-
puting a low-rank matrix and reconstructing the extracted query and support features. It enforces relevant latent structure and alignment between query and support features. Finally, [241] introduces a loss function to promote orthogonality between classes in the feature space.

### 3.2.4 Meta-Learning

While Meta-Learning methods are very common for FSC, they are much rarer in the FSOD literature. It is explained easily by the difficulty of the task and the complexity of meta-learning approaches. They often require training a meta-learner model for generating weights or gradient updates to a smaller classification network. However, detection models are much larger than the classification ones, which makes the meta-learning approaches impractical for FSOD. Nevertheless, the FSOD literature borrows some techniques from the meta-learning field. In particular, most attention and metric-learning-based methods for FSOD are trained using an episodic training strategy (see Tab. 3.1). In addition, many works are presented within the meta-learning perspective because of the episodic training, here are some examples: Meta Faster R-CNN [231], Meta R-CNN [288], Meta-DETR [232], and GenDet [250].

Yet, there are a few attempts at solving FSOD with meta-learning approaches. MetaDet [276] extends Faster R-CNN with the MAML framework. Specifically, they choose to generate weights only for the detection head of Faster R-CNN, considering that the feature extractor and RPN are class-agnostic and do not need adaptation. This significantly reduces the size of the generated weights and makes it possible to use MAML. During base training, only the detector is trained on the base dataset. Then a fine-tuning phase occurs, the meta-learner is trained to predict the weight of the detection head only provided with support examples for the novel classes. At the same time, the detector is fine-tuned on the support set, with all class-agnostic parts frozen. At test time, the meta-learner predicts weights for the novel classes to extend the detector's head and the detector can be used as a regular detector. Similarly, Sylph [277] applies the same idea but only to the classification branch of the detector, assuming that the regression is also class-agnostic.

This section draws an almost exhaustive list of the contributions to the FSOD field (see Tab. 3.1). As for FSC, several research tracks explore FSOD. However, meta-learning is a lot less popular approach for detection compared to classification. Instead, attention-based methods (often trained with an episodic strategy) are the mainstream approaches. Nevertheless, there is no consensus about the best way to tackle FSOD, and fine-tuning or metric-learning contributions are often simpler and still competitive.

## 3.3 Few-Shot Object Detection on Remote Sensing Images

Few-Shot Object Detection is a relatively recent field in computer vision and so far, it has been applied mostly to natural images and in particular on Pascal VOC and MS COCO datasets (Sec. 3.5 explains how they are prepared for the few-shot setting). However, there are a few contributions that apply FSOD techniques on Remote Sensing Images (RSI), these are highlighted in green in Tab. 3.1.
RSI are notoriously more challenging than natural images for the detection task. Objects are smaller and more numerous, they can be arbitrarily oriented, and the background is often more complex. Therefore, object detection methods applied to RSI often comprise some tricks to better deal with the specificities of RSI. FSOD techniques applied to RSI follow the same trend. Among these tricks, the use of multiscale features is certainly the most common. For instance, FSOD-RSI [223] extends FRW with three levels features map to better deal with small objects. Similarly, [230, 245, 269] leverage multiscale features for either the query image, support images or both. FSOD applied to RSI is based on attention-mechanisms [223, 230, 245, 238, 240, 246] or fine-tuning strategies [261, 266, 269, 289], but to our knowledge, there is no FSOD method based completely on a metric-learning approach.

Among these contributions, some tackle the detection problem with other modalities than visible light. Indeed, this is a problem of interest as earth observation is often conducted with non-visible light. Image quality is highly dependent on the weather conditions, and half the earth at night is unobservable with visible light. Therefore, a lot of applications rather use infrared light or Synthetic-Aperture Radar (SAR). Two articles tackle few-shot detection in SAR images [240, 245], yet without any notable adjustment to account for the modality change.

These contributions are of particular interest for COSE as the goal is to design efficient detection methods for high-resolution images. Some extensions for the CAMELEON project are already planned with multi-spectral images and LIDAR. Hence, methods able to adapt from one modality to another are especially valuable.

## 3.4 Extension of the Few-Shot Object Detection Setting

As for Few-Shot Classification, the Few-Shot Object Detection setup has several extensions. These settings are more challenging but reflect better real-life use cases.

### One-shot and Zero-shot Object Detection

First, in the case of extremely limited annotations, object detection is still achievable. One-Shot OD has been addressed by several works [227, 221, 236, 242] that we present in the above section. These approaches are not different from the few-shot setting, it simply is more difficult. However, in the zero-shot setting, it becomes even more challenging as no image example is available for the novel classes. The common approach in this setting is to leverage semantic representations from the class labels and condition the detection on this information [290]. Recently introduced large language-visual models such as CLIP [291] provide strong improvements for various zero-shot tasks and object detection is no exception. For instance, [279] trains a prompt generator in a meta-learning fashion to condition the detection on novel classes. Alternatively, DINO [68] and DETReg [267] conceive strong self-supervised pre-training schemes specifically adapted for object detection, which translate into impressive performance in a low shot setting. Finally, [278] leverage a transductive pseudo-labeling approach to improving zero-shot detection. To our knowledge, this is the only transductive method applied to few-shot detection.
Generalized and Incremental FSOD

The goal of FSOD is to adapt to novel classes; however, in many cases, the performance on base
classes matters as well. This is the case in the Generalized FSOD (G-FSOD) setting where we are
interested in detecting both base and novel classes. The incremental setting extends G-FSOD with
several adaptations to novel classes without forgetting the previously seen classes. For G-FSOD, a
naive approach is to train two detectors: one on base classes and one on all classes (base and novel),
as a fine-tuned version of the first one. Outputs from both detectors are combined at test time to
achieve better performance on base and novel classes. This is adopted by [259] with two Faster R-
CNN. However, other contributions propose more sophisticated methods. CFA [268], for instance,
proposes a regularization loss to prevent forgetting base classes. Alternatively, [273] duplicates the
detection head to process separately the foreground and background samples and prevents classifi-
cation bias toward base classes. In the incremental setting, the naive approach from [259] does not
scale as it would require duplicating the detector each time novel classes are added. Instead, [228,
277] train a meta-network to generate classifier weights for novel classes on-the-fly unlocking con-
venient adaptation. Incremental DETR [275] adopts a different strategy based on fine-tuning and
distillation to prevent forgetting previously seen classes.

Cross-domain Few-Shot Object Detection

Last but not least, Cross-Domain FSOD (CD-FSOD) tackles the few-shot object detection task in the context of domain adaptation. CD-FSOD aims at designing methods able to generalize to new kinds of images. Just as for classification, two sub-tasks have been explored in the literature: CD-FSOD with and without class shift. For COSE, both tasks are relevant but CD-FSOD with class shift precisely corresponds to their application. Indeed, once a system is in operation, it will likely encounter new objects and domains. While the images will always be taken from above, their general aspect may change a lot due to weather conditions, different landscapes or carrier altitude. Therefore, solving CD-FSOD with class shift is crucial for the CAMELEON system.

However, this field remains barely untouched. To our knowledge, only a few contributions tackle CD-FSOD. First, without class shift, several works address this problem with augmentation-based approaches. The idea is to leverage the few target examples to augment the source images so that they become plausible samples from the target domain. For instance, [281] proposes a directive data augmentation procedure that optimally augments the source examples, so their features are close to the features of the target examples. The detector is then trained as a regular detector on the augmented examples. Likewise, [292, 280] propose source-to-target translation networks that convert source images into target images. These networks are trained adversarially with a discriminator that aims to distinguish between domains. Closely related, Cross-Domain CutMix [282] crafts an augmentation technique that mixes two domains by cropping and pasting objects from the target domain into source images and vice-versa.
The methods discussed above assume that the source and target domains share the same label space, i.e., they have the same classes. This assumption allows for building simple domain augmentation approaches as once the source images are translated into the target domain, their annotations can be directly used for training. When classes shift between source and target domains, this is no longer an option. In this very challenging setting, most FSOD methods showcase poor results and adaptation capabilities. Naive fine-tuning is often the best alternative in this case. To our knowledge, there are still no contributions that tackle this task. Only two articles [283, 284] pave the way for future research in this direction with dedicated benchmarks and baselines. Both propose to study the generalization capabilities of FSOD methods on various datasets after a large base training on COCO dataset. First, [283] combines three datasets: 1) ArTaxOr [293], a close-up insect images dataset, 2) UODD [294], an underwater image dataset, and 3) DIOR. The authors propose a simple self-distillation strategy, similar to self-supervised approaches, during the fine-tuning on the new domains. Second, [284] builds a more complete benchmark called Multi-dOmain FSOD (MoFSOD) with 10 different target domains. However, the authors study the influence of two different source datasets COCO and LVIS [96]. They also provide a domain distance measure that assesses the similarity between a dataset and COCO. This measure is the recall of a detector trained on COCO applied to a dataset in a class-agnostic manner. Intuitively, if a dataset is close to COCO (in terms of classes and aspects), the trained detector will detect a lot of objects (even if the classes are wrong) and will have a high recall. Based on this similarity measure they study the impact of freezing some layers of the detector during the fine-tuning. Previous works recommend only fine-tuning the detection head while keeping the backbone frozen. However, [284] shows that this is true only for sufficiently similar datasets. In other words, when the source-target gap is large, it is better to fine-tune the model entirely for better adaptation. Unfortunately, the authors did not provide an easy-to-use meta-dataset for future research and addressing CD-FSOD remains challenging due to complex initial data processing.

## 3.5 Dataset preparation and evaluation in the Few-Shot setting

### 3.5.1 Adapting detection datasets in the Few-Shot setting

There are no specific datasets for Few-Shot Object Detection. Instead, regular detection datasets can be adapted to the few-shot setting. In this section, we describe this process in the case of the four datasets on which this PhD project mainly focuses: DOTA [77], DIOR [95], Pascal VOC [5] and COCO [6].

The conversion of a dataset for the *N*-ways *K*-shots setting is straightforward. First, the set of classes is divided into two sets: the base and novel class sets (with $|C_{\text{novel}}| = N$). The class split for each dataset is fixed by common practices (for Pascal VOC and COCO) or taken at random (for DOTA and DIOR) when no convention is set in the literature. Tab. 3.2 gives the class split that will be used throughout this PhD thesis. Then, the instances of the novel classes are filtered from the dataset to keep only *K* images per novel class. This filtering operation is performed in two steps:
1. for each novel class $c \in C_{\text{novel}}$, K images containing at least one instance of class c are selected as the support examples. They constitute the support set.

2. the instances of the novel classes are removed from all other images in the dataset.

This choice is motivated by the presence of *distractors*. This concept and further explanations about how they influence the few-shot training in detection will be presented in Sec. 4.1. It is important to note that as FSOD is a recent field, several preparation techniques coexist in the literature. However, the one described above seems the most reasonable and common in current FSOD works.

Nevertheless, the existence of various preparation settings makes the comparison with existing
methods difficult as this choice is rarely discussed in the articles. It may be challenging to fig-
ure out precisely how the datasets were prepared, not to mention the choice of the training strategy
(i.e., episodic or direct fine-tuning). The reported FSOD performance in the literature should then
be regarded with a critical eye.

**3.5.2 Evaluation protocol for Few-Shot Object Detection**

The common practice in FSC is to randomly sample a support set from the training split of the
whole dataset, adapt the model with it (through fine-tuning or direct adaptation), and finally, make
the predictions and compute the relevant metrics on the test split. This is repeated many times
with different support sets and the scores are finally averaged to give a robust evaluation of the
generalization capabilities of the models.

For detection, the same principle should be applied to get a robust assessment of the models’ per-
formance. However, the adaptation of such models is often quite long compared to classification
models. Indeed, detection models are much larger than classification ones, thus they take more
time to adapt to novel classes. Before going further, we need to distinguish two approaches, on the
one hand, the fine-tuning strategy and on the other hand all other strategies (i.e., metric learning,
meta-learning and attention-based methods). The main distinction is that the latter use the support
set during inference whereas fine-tuning approaches only leverage it during the second phase of
training. Fig. 3.4 illustrates the two different approaches for FSOD model evaluation and exhibits a
time estimation for training and evaluating one model, following the general recommendations of
FSC (i.e., at least 100 repetitions with various support sets).

**Evaluation of fine-tuning FSOD approaches**

Repeated evaluation requires fine-tuning the base model (i.e., the model after base training) with various support sets. Fine-tuning FSOD methods can take up to a few hours and repeated evaluation may take days <fnref n="1" />, which is not practical. A reasonable compromise is to perform a limited number of runs (between 10 and 30), which is sufficient according to empirical studies in [258]. Even though, robust evaluation is still an intensive process in this setting.

<footnote n="1">¹a typical setup in FSC is to repeat 100 times the adaptation, even if an FSOD model takes only 30 minutes to adapt to the novel classes through fine-tuning, the robust evaluation would take almost 2 days.</footnote>
<table><caption>Table 3.2: Base / Novel class splits for the different datasets used throughout this thesis. The novel classes in the COCO dataset correspond to all classes in Pascal VOC.</caption><thead><tr><th></th><th>Novel classes</th><th>Base classes</th></tr></thead><tbody><tr><td>Pascal VOC</td><td>bird, bus, cow, motorbike, sofa</td><td>aeroplane, bicycle, boat, bottle, car, cat, chair, diningtable, dog, horse, person, pottedplant, sheep, train, tvmonitor</td></tr><tr><td>MS COCO</td><td>person, bicycle, car, motorcycle, airplane, bus, train, boat, bird, cat, dog, horse, sheep, cow, bottle, chair, couch, potted plant, dining table, tv</td><td>truck, traffic light, fire hydrant, stop sign, parking meter, bench, elephant, bear, zebra, giraffe, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket, wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake, bed, toilet, laptop, mouse, remote, keyboard, cell phone, microwave, oven, toaster, sink, refrigerator, book, clock, vase, scissors, teddy bear, hair drier, toothbrush</td></tr><tr><td>DOTA</td><td>storage-tank, tennis-court, soccer-ball-field</td><td>plane, ship, baseball-diamond, basketball-court, ground-track-field, harbor, bridge, small-vehicle, large-vehicle, roundabout, swimming-pool, helicopter, container-crane</td></tr><tr><td>DIOR</td><td>airplane, baseball field, tennis court, train station, wind mill</td><td>airport, basketball court, bridge, chimney, dam, expressway service area, expressway toll station, golf course, ground track field, harbor, overpass, ship, stadium, storage tank, vehicle</td></tr></tbody></table>
<figure><img src="image_53.png" /><figcaption>Figure 3.4: Illustration of two evaluation processes existing in the literature. One for fine-tuning methods (left) and one available for all other methods (right).</figcaption></figure>

## Evaluation of other FSOD approaches

Other methods adapt to novel classes at inference time given a support set. Therefore, they can be more robustly evaluated (in a reasonable time) than their fine-tuning counterparts. Adaptation is often fast compared to the fine-tuning phase and can be more easily repeated. However, detection models that are based on metric learning or attention still require a fine-tuning phase at least for the regression branch. A support set must be used for this as well and its choice influences the performance of the model, even if adaptation is repeated multiple times after the fine-tuning. Yet this fine-tuning step is often even more time-consuming than basic fine-tuning approaches as the models are augmented with costly adaptation modules. In this case, the common setting is to repeat the adaptation at inference multiple times after only a single fine-tuning of the model. These settings provide a sweet spot between evaluation robustness and computation time. They will be employed in our all experiments unless specified, both for fine-tuning, metric learning and attention-based approaches.

## 3.6 Conclusion

In this chapter, we reviewed the FSOD literature. This field is relatively recent and fastly growing. It has significantly evolved since the beginning of this project. In 2020, most FSOD works were based on attention-based approaches, yet fine-tuning techniques are now getting more and more interest. This review helps to understand the main directions that have already been explored and the relevant tracks that need to be pursued.
CHAPTER 4

UNDERSTANDING THE CHALLENGES OF FEW-
SHOT OBJECT DETECTION

ABSTRACT

The detection task becomes extremely challenging when limited annotated data is available. In this chapter, we explore the reasons behind this difficulty. In particular, we focus on the case of aerial images for which it is even harder to apply FSOD techniques. It turns out that small objects are especially challenging for the FSOD task and are the main source of poor performance in remote sensing images.

<figure><img src="image_54.png" /></figure>

P. Le Jeune and A. Mokraoui, "Improving Few-Shot Object Detection through a Performance Analysis on Aerial and Natural Images," 2022 30th European Signal Processing Conference (EUSIPCO), Belgrade, Serbia, 2022, pp. 513-517, doi: 10.23919/EUSIPCO55093.2022.9909878.

<figure><img src="image_55.png" /></figure>

P. Le Jeune and A. Mokraoui, "Amélioration de la détection d'objets few-shot à travers une analyse de performances sur des images aériennes et naturelles." GRETSI 2022, XXVIIIème Colloque Francophone de Traitement du Signal et des Images, Nancy, France

CONTENTS

<table><tr><td>4.1 Distractors in the Few-Shot Data Regime</td><td>78</td></tr><tr><td>4.2 The Increased Challenge of Aerial Images.</td><td>78</td></tr><tr><td>4.3 Conclusion</td><td>83</td></tr></table>

In this chapter, we present our first contribution to the FSOD field. Specifically, this section presents an analysis of the difficulties of going from a regular to a few-shot data regime for the detection task, especially for aerial images.
## 4.1 Distractors in the Few-Shot Data Regime

First, changing the set of classes of interest during the training procedure (i.e., between base training and fine-tuning) is problematic. Classes considered as background can become objects of interest, which goes against the knowledge acquired during the base training phase. This is embodied by the concept of *distractors*, introduced in [263]. It refers to examples that provide wrong supervision to a model during training. We choose to refine this concept in two categories: *self-distractors* and *co-occurrence distractors*. Self-distraction occurs when annotated and non-annotated instances of a class are visible in the same image. The non-annotated instances are called self-distractors. This can happen when there are annotation mistakes in a dataset, but it can also happen in the few-shot settings. For instance, if annotations of support examples are filtered (e.g., to keep only one annotation per support image), all the filtered instances will become self-distractors during fine-tuning. When fine-tuning on the support examples, the annotated instances of the novel classes will provide correct supervision to the model. However, the non-annotated instances will be considered as background and wrong supervision will be propagated in the model. This explains why it is more sensible to keep all annotations of the novel classes in the support set even though it does not fully comply with the original *N*-ways *K*-shots setting. In the literature, this choice is barely discussed and early works in the field employ either the strict one annotation per image sampling or the *self-distractor-free* sampling described above. Using self-distractor-free sampling often results in improved performance; however, no analysis was conducted to explain the origins of these gains (either coming from more examples or thanks to more coherent supervision). In our experiments, both setups were used as this issue was encountered in the middle of this project. We will clearly specify what setting is used for all our experiments.

The second type of distraction, *co-occurrence distraction*, happens when novel class instances are visible in images during base training. Their annotations have been filtered out, therefore they are considered as background. Of course, this makes sense during base training as the novel classes are by definition unknown at this point. However, in this setup, the model is specifically trained to consider these objects as background, whereas if no co-occurrences of the novel and base classes were allowed, no background supervision would be given to novel class instances. This could be achieved by removing all images containing such co-occurrences from the dataset. However, this type of distractor is much less frequent than the self-distractors (see Fig. 4.1). In addition, these mostly occur during the base training phase, and even if they provide incorrect supervision, fine-tuning will rectify it. Therefore, we choose to keep images with co-occurrence distractors during base training.

## 4.2 The Increased Challenge of Aerial Images

The difficulties described in the previous section are not specific to any kind of image. However, it appears from the scarce literature and our experiments that applying FSOD on aerial images is much more difficult than on natural ones.
<figure><img src="image_56.png" /><figcaption>Figure 4.1: Co-occurrences between the classes of the four datasets of interest DOTA, DIOR, Pascal VOC and MS COCO. Novel classes are highlighted in red. For MS COCO only novel class labels are shown for clarity.</figcaption></figure>
<table><caption>Table 4.1: Regular baseline performance (mAP with a 0.5 IoU threshold) on DOTA, DIOR and Pascal VOC datasets (i.e., trained with all annotations). The baseline model is FCOS [45], trained on all classes (base and novel) and with all available annotations in each dataset. Then the mAP is computed on base and novel classes separately.</caption><thead><tr><th colspan="2">DOTA</th><th colspan="2">DIOR</th><th colspan="2">Pascal VOC</th></tr></thead><tbody><tr><td>Base Classes mAP</td><td>Novel mAP</td><td>Base mAP</td><td>Novel mAP</td><td>Base mAP</td><td>Novel mAP</td></tr><tr><td>60.87</td><td>69.69</td><td>72.82</td><td>81.48</td><td>65.47</td><td>68.02</td></tr></tbody></table>

At the beginning of this PhD, only very few works addressed FSOD in aerial images. Among those, *Few-Shot Object Detection via Feature Re-Weighting* (FRW) [225] and *Few-Shot Object Detection With Self-Adaptive Attention Network for Remote Sensing Images* (WSAAN) [223] were the most popular. These two works were not evaluated on the same datasets, preventing any useful comparison. In addition, many architectural choices differ from one another, for instance, the underlying detection frameworks and the backbones. Therefore, we choose to implement these methods within a single framework, preventing most architectural discrepancies. Our proposed framework will be described thoroughly in Chap. 6. We also re-implemented Dual AwareNess Attention (DANA) [234] as it was one of the best-performing methods on COCO dataset at that time. We analyze here the behaviors of these three FSOD techniques both on natural and aerial images. The main idea is to compare the performance of the three methods in regular and few-shot data regimes. The regular data regime corresponds to the vanilla detector (i.e., without any modification for the few-shot setting) and with full access to the novel class annotations in the dataset (i.e., no annotation filtering). In our re-implementation of FRW, WSAAN and DANA, the underlying detector is FCOS [45]. Thus, the regular baseline is an FCOS detector trained on the full datasets. To conduct this experiment, we only select DOTA, DIOR and Pascal VOC as they have roughly the same number of classes. COCO however has 4 times more classes which brings additional complexities. The performance results of the regular baseline are available in Tab. 4.1. Specifically, FCOS is trained on each dataset with full access to the annotations for both base and novel classes. Then the mAP is computed on each class individually and averaged on base and novel classes separately. This gives an overview of the performance of the model respectively on base and novel classes in a regular data regime.

It seems tempting here to extrapolate the FSOD performance on DOTA and DIOR from the performance on Pascal VOC. The regular baseline (FCOS) achieves similar performance on these two datasets, which contain the same number of classes and roughly the same number of images. Thus, one could have expected close FSOD performance on these datasets. This is quite different from the actual results reported in Tab. 4.2. The FSOD performance on DOTA and DIOR is significantly lower compared with the results on Pascal VOC. To better visualize this finding, Fig. 4.2 represents the few-shot performance as dark bars while regular baseline performance as lighter rectangles. The height of the rectangle is set as the mAP on either the base or novel classes (in blue and red respectively). This clearly illustrates the different behaviors of FSOD methods applied on aerial or natural images.
<table><caption>Table 4.2: Comparison of mAP<sub>0.5</sub> of several methods on DOTA, DIOR and Pascal VOC datasets. For each method, mAP is reported for different numbers of shots $K \in \{1, 3, 5, 10\}$ and separately for base and novel classes. Blue and red values represent the best performance on base and novel classes respectively, for each dataset.</caption><thead><tr><th rowspan="2"></th><th colspan="4">DOTA</th><th colspan="4">DIOR</th><th colspan="4">Pascal VOC</th></tr><tr><th>FRW</th><th>WSAAN</th><th>DANA</th><th>FRW</th><th>WSAAN</th><th>DANA</th><th>FRW</th><th>WSAAN</th><th>DANA</th><th>FRW</th><th>WSAAN</th><th>DANA</th></tr></thead><tbody><tr><th>K</th><td>Base</td><td>Novel</td><td>Base</td><td>Novel</td><td>Base</td><td>Novel</td><td>Base</td><td>Novel</td><td>Base</td><td>Novel</td><td>Base</td><td>Novel</td></tr><tr><th>1</th><td>47.24</td><td>13.35</td><td>45.55</td><td>12.19</td><td>49.78</td><td>12.52</td><td>56.67</td><td>16.92</td><td>56.41</td><td>15.48</td><td>58.78</td><td>20.64</td></tr><tr><th>3</th><td>46.50</td><td>25.32</td><td>44.18</td><td>24.42</td><td>49.67</td><td>20.70</td><td>58.05</td><td>25.08</td><td>51.72</td><td>13.84</td><td>59.14</td><td>27.26</td></tr><tr><th>5</th><td>48.60</td><td>29.57</td><td>47.56</td><td>31.44</td><td>53.49</td><td>24.96</td><td>60.75</td><td>32.58</td><td>60.79</td><td>30.32</td><td>62.12</td><td>34.16</td></tr><tr><th>10</th><td>48.52</td><td>37.10</td><td>46.72</td><td>35.12</td><td>53.25</td><td>34.39</td><td>61.47</td><td>35.56</td><td>61.88</td><td>33.41</td><td>62.49</td><td>36.43</td></tr></tbody></table>

<figure><img src="image_57.png" /><figcaption>Figure 4.2: Performance comparison between FCOS trained in a regular data regime versus three few-shot baselines, FRW, WSAAN and DANA (all based on FCOS as well) on three datasets: DOTA, DIOR and Pascal VOC.</figcaption></figure>

It is generally irrelevant to compare the performance of a method from one dataset to another, especially with images of different natures. Each dataset has its own characteristics (resolution, intra-class variety, color range, etc.) and therefore a given model will not perform equally well on two distinct datasets according to a pre-defined performance metric. Hence, we cannot compare the absolute performance of a FSOD method on Pascal VOC and DOTA and the previous extrapolation is not valid. Nevertheless, there is a pattern: FSOD methods work consistently better on natural images compared to aerial images. To understand this phenomenon, we need a way to fairly compare the FSOD performance across several datasets. To this end, we propose to look at the relative performance of the FSOD methods against the regular baseline (i.e. FCOS in our case) using the following metric:

$$RmAP = \frac{mAP_{FSOD} - mAP_{Baseline}}{mAP_{Baseline}}. \qquad (4.1)$$
RmAP assesses how well a FSOD method is performing on different datasets compared with the regular detection performance. Hence, it represents how much performance is lost when switching from the regular to the few-shot regime. This is exactly what is illustrated in Fig. 4.2, white percentages are RmAP values. RmAP is significantly lower on DOTA and DIOR compared to Pascal VOC, both for base and novel classes. This way, we can quantitatively confirm the intuition emerging from Tab. 4.2: FSOD works better on natural images.

We hypothesize that this performance gap is mainly due to differences in the object sizes within the datasets. In aerial images, objects are much smaller on average. This is already an issue for object detection: small objects are challenging to find. The paradigm of current vision models is to have deep feature representations with increasing fields of view. The Field of View (FoV) of a specific layer is the area in the input image that influences the value of one location in the feature map of that layer. In deeper layers, the FoV is often quite large compared to small objects' size and object features are diluted with their irrelevant and noisy surroundings. Thus, it reduces the activation strength at the object location, and the object can easily be missed. Feature Pyramidal Networks [1] and various other tricks were introduced to solve this issue, as discussed in Sec. 2.1.3.4. However, this problem is largely amplified for FSOD. It is still difficult to detect small objects, but in addition, they are poor examples for adapting the model (either through fine-tuning or direct adaptation).

<figure><img src="image_58.png" /><figcaption>Figure 4.3: Box plot of objects size in DOTA, DIOR and Pascal VOC and MS COCO. On the left side, boxes represent the overall size distribution in each dataset. On the right side, the distributions are split by class and ordered by average size. As MS COCO contains 80 classes, we choose not to include the per class box plots for it in this plot.</figcaption></figure>
To support this hypothesis, we first conduct a brief size analysis of the four datasets DOTA, DIOR, Pascal VOC and MS COCO (see Fig. 4.3). Aerial datasets contain far smaller objects than natural ones. Plus, in aerial datasets, the size of objects in different classes differs a lot. Some classes contain only small objects, while others only large objects. In Pascal VOC, this class' size variance is limited. We argue that it is more difficult for the model to extract relevant information from small support examples but also to learn more diverse features to deal with greater objects' size variance. Incidentally, this partly explains the greater difficulty of MS COCO. To support this claim, we conduct a per-class performance analysis on DOTA, DIOR and Pascal VOC. The results of this comparison are available in Fig. 4.4. In this figure, the performance is reported per class against the average size of the class. The first row reports absolute mAP values (with 0.5 IoU threshold) both for FRW and FCOS (baseline). In the second row, the mAP gap between the FRW and the baseline is plotted against the objects' size. We did not report RmAP values for the sake of visualization. RmAP can take large values (e.g. when the regular baseline mAP is low) and this squeezes the interesting part of the plot in a narrow band around 0. Larger objects are easier to detect. It is true in both data regimes, but this trend is reinforced in the few-shot regime (in the first row, the blue trend lines are steeper than the black ones). This is observed for base classes but not always for novel classes, probably because the trends on novel classes are not reliable due to the limited number of points. Fig. 4.5 shows a more reliable trend for novel classes when the results from the three datasets are aggregated. Finally, the few-shot methods, which leverage support information to condition the detection can surpass the baseline in some cases. All three methods here are attention-based, and therefore, benefit from having support examples available during inference to condition the detection. This would not be the case with fine-tuning approaches. However, this seems advantageous only when the objects are large. On the contrary, when the objects are small, the performance is degraded. It confirms that small objects are poor examples to condition the detection on. For novel classes; however, the performance is always below the baseline, even if the gap shrinks with larger objects. This is expected as the network only received weak supervision for these classes. This comparative analysis confirms that detecting small objects is a very difficult task in the few-shot regime. It is hard to extract useful information from small support objects. Even worse, this information can be detrimental for the detection. Existing FSOD methods are not designed to deal with small objects, hence the application of these methods on aerial images does not yield satisfactory results. It is therefore crucial to develop FSOD techniques that target specifically small objects. Incidentally, we will address this point in Sec. 6.3 and in Chap. 8.

## 4.3 Conclusion

In this chapter, we presented our first contribution to the FSOD field with an analysis of the challenges raised by the few-shot regime for the detection task. These difficulties are reinforced when FSOD is applied to aerial images as they contain smaller objects. This gives a clear direction for this PhD project: improving the handling of small objects in FSOD methods. To this end, we dedicate Sec. 6.3 and the entire Part III of this thesis.
<figure><img src="image_59.png" /><figcaption>Figure 4.4: Performance comparison between FRW baseline – with 10 shots – (blue and red dots) and regular baseline (black stars) on three different datasets: DOTA, DIOR and Pascal VOC. (top) Mean average performance of the two methods plotted per class against average object size. (bottom) gap between FRW baseline and regular baseline, per class. Positive values indicate better performance than the regular baseline.</figcaption></figure>
<figure><img src="image_60.png" /><figcaption>Figure 4.5: Comparison of FRW (FSOD) and FCOS (baseline) performance against object size on the three datasets DOTA, DIOR and Pascal VOC together. (top) absolute mAP<sub>0.5</sub> values. (bottom) RmAP computed against the regular baseline. The RmAP plot has been cropped for visualization reasons. Only the class container-crane from DOTA dataset is not visible (with a RmAP of 150%).</figcaption></figure>
# PART II

## IMPROVING FEW-SHOT OBJECT DETECTION THROUGH VARIOUS APPROACHES
CHAPTER 5

EXPERIENCE FEEDBACK ABOUT METRIC LEARN-
ING FOR FSOD

ABSTRACT

Prototypical Faster R-CNN (PFRCNN) is a novel approach for FSOD based on metric learning. It embeds prototypical networks inside the Faster R-CNN detection framework, specifically in place of the classification layers in the RPN and the detection head. PFRCNN is applied to synthetic images generated from the MNIST dataset and to real aerial images with DOTA dataset. The detection performance of PFRCNN is slightly disappointing but sets a first baseline on DOTA. However, the experiments conducted with PFRCNN provide relevant information about the design choices for FSOD approaches.

<figure><img src="image_61.png" /></figure>

P. L. Jeune, M. Lebbah, A. Mokraoui and H. Azzag, "Experience feedback using Representation Learning for Few-Shot Object Detection on Aerial Images," 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Pasadena, CA, USA, 2021, pp. 662-667, doi: 10.1109/ICMLA52953.2021.00110.

CONTENTS

<table><tbody><tr><td>5.1</td><td>Motivation and Principle.</td><td>90</td></tr><tr><td>5.2</td><td>Prototypical Faster R-CNN for FSOD</td><td>90</td></tr><tr><td>5.2.1</td><td>Extending Faster R-CNN with Prototypical Networks.</td><td>91</td></tr><tr><td>5.2.2</td><td>Training Procedure</td><td>93</td></tr><tr><td>5.2.3</td><td>Iterative improvements</td><td>95</td></tr><tr><td>5.3</td><td>Performance on Artificial Data</td><td>96</td></tr><tr><td>5.3.1</td><td>MNIST-LOC Dataset</td><td>96</td></tr><tr><td>5.3.2</td><td>Implementation details</td><td>97</td></tr><tr><td>5.3.3</td><td>Detection performance on MNIST-LOC</td><td>97</td></tr><tr><td>5.4</td><td>Difficulties on Aerial Images.</td><td>98</td></tr><tr><td>5.5</td><td>Insights and conclusion</td><td>102</td></tr></tbody></table>
As a first step into the Few-Shot Object Detection field, we proposed a naive approach to solve the detection task in the few-shot regime. To give some context, at the beginning of this project, FSOD was a very recent domain and very few articles tackled this challenging task, especially applied to aerial images. Therefore, we took inspiration from the Object Detection and Few-Shot Classification literature embodied respectively by Faster R-CNN [33] and Prototypical Networks [126]. This chapter presents Prototypical Faster R-CNN, our first attempt at solving FSOD. We begin by presenting the motivation behind this contribution and its main principle. Then, the training procedure and several tricks are proposed to improve training stability and detection quality. Finally, Prototypical Faster R-CNN is applied to synthetic and aerial images to assess its generalization capabilities and understand its limitations.

## 5.1 Motivation and Principle

In 2020, most of the FSOD literature was focused on attention-based approaches (see Tab. 3.1); however, the simplicity and success of the metric learning classification models was tempting. Thus, we proposed Prototypical Faster R-CNN (PFRCNN), an extension of Faster R-CNN based on metric learning. The key idea is to replace the classification layers from Faster R-CNN (i.e., in the Region Proposal Network (RPN) and in the Classification head) with prototypical networks. It is similar to RepMet [251] that leverages class-representative vectors in the classification head. However, there are two major differences with PFRCNN. First, RepMet only replaces the classification layer in the second stage of Faster R-CNN, not in the RPN. Hence, the adaptation to novel classes is only done in the second stage. Even if the RPN is presented as a class-agnostic detector, it specializes in the classes seen during training. As only base classes are annotated during the first phase of training, objects from novel classes will be filtered out by the RPN, leaving no chance for the second stage to detect them. Even if it is trained to have a high recall, the RPN will mostly generate proposals on base classes, which is harmful in a few-shot regime. Second, RepMet learns the class-generative vectors from fine-tuning on the few available examples of the novel classes. Instead, a prototypical network computes its *prototypes* directly from the few available examples. Finally, Prototypical Networks can adapt to novel classes without any fine-tuning. Hopefully, this property would transfer to Faster R-CNN by replacing its classification layer with such malleable modules. For COSE's application, this would be ideal as the detection model could adapt “on the fly” at a low cost.

## 5.2 Prototypical Faster R-CNN for FSOD

Before explaining in detail how the prototypical networks can be embedded into Faster R-CNN, let us define a few notations and detail the functioning of Faster R-CNN. The backbone, RPN and detection head are respectively denoted as *f*, *g*, and *h*. The backbone extracts feature $F_q$ from the input – or query – image $I_q$:

$$f(I_q) = F_q. \qquad (5.1)$$
The backbone extracts features at multiple scales using an FPN, but for simplicity, we regroup all these features into one notation: $F_q$. The backbone is a ResNet-50, the FPN extracts features from 3 different levels with respective strides 8, 16, and 32. The RPN takes $F_q$ as input and computes both proposals boxes $\bar{b}_i$ and objectness scores $o_i$ for all locations in the feature maps:

$$g(F_q) = \{(\bar{b}_i, o_i)\}_{i=1}^{M}, \qquad (5.2)$$

where $M$ is the number of generated boxes. It changes with the number of anchor boxes defined per location, in our case, it is set to 3. Hence, $M$ is three times the number of locations in the feature map. Then, the top 1000 boxes with the highest objectness scores are selected to extract the proposals features $\xi_i$ with the RoI Align layer:

$$\text{RoIAlign}(F_q, \bar{b}_i) = \xi_i. \qquad (5.3)$$

Finally, the detection head $h$ outputs classification scores for each proposal from its features and refines its box coordinates:

$$h(\xi_i) = \hat{y}_i = (\hat{b}_i, \hat{l}_i), \qquad (5.4)$$

where $l_i \in [0, 1]^{|C|+1}$ is a vector of classification scores. There is one more element in $l_i$ than in $C$ because Faster R-CNN deals with background as a class.

### 5.2.1 Extending Faster R-CNN with Prototypical Networks

To replace the classification layer in Faster R-CNN by prototypical networks, we propose to change the output dimension of the last layer in the classification branches of both the RPN and the head. That way, instead of producing a classification (or objectness) score per box, these networks output embedding vectors. Each vector represents the information contained inside the corresponding box. We denote these embedding vectors of the RPN and the classification head $z_i^{\text{RPN}}$ and $z_i^{\text{head}}$ respectively. Their dimension is set to 128 ($z_i \in \mathbb{R}^{128}$) and is kept fixed in all our experiments. Hence, the outputs of the RPN and the detection head become:

$$g(F_q) = \{(\bar{b}_i, z_i^{\text{RPN}})\}, \qquad (5.5)$$

$$h(\xi_i) = (\hat{b}_i, z_i^{\text{head}}). \qquad (5.6)$$

Then, the objectness and classification scores for each proposal are computed with prototypical networks based on class prototypes computed from support examples. Prototypes are computed from the support set $\{(I_k^c, b_k^c)\}_{1 \le k < K}$. Specifically, each support image is fed into the backbone to $c \in C_{\text{novel}}$.
<figure><img src="image_62.png" /><figcaption>Figure 5.1: Illustration of the architecture of Prototypical Faster R-CNN.</figcaption></figure>

extract its features $F_k^c$ and then the example features are extracted with RoI Align:

$$z_{k,c}^{\text{RPN}} = g(F_k^c), \qquad (5.7)$$

$$\Phi_k^c = \text{RoIAlign}(z_{k,c}^{\text{RPN}}, b_k^c), \qquad (5.8)$$

$$z_{k,c}^{\text{head}} = h(F_k^c), \qquad (5.9)$$

$$\Psi_k^c = \text{RoIAlign}(z_{k,c}^{\text{head}}, b_k^c). \qquad (5.10)$$

This gives RPN features and classification features for each support image, denoted $\Phi_k^c$ and $\Psi_k^c$ respectively. Note a slight abuse of notation here, when only the embedding part of $g$ and $h$ is used to project the features extracted by the backbone (i.e., not the regression part). When multiple examples are available for a class (i.e., $K \ge 1$), their embeddings are averaged to get one prototype per class:

$$\Phi^c = \frac{1}{K} \sum_{k=1}^{K} \Phi_k^c, \qquad (5.11)$$

$$\Psi^c = \frac{1}{K} \sum_{k=1}^{K} \Psi_k^c. \qquad (5.12)$$
For each proposal, we compute the classification score for class c as the likelihood of the region in the input image representing an object of class c. To do so, we suppose that the class distributions over the embedding space are Gaussian distributions centered on the class prototypes. Hence, the classification score of the proposal i for class c is:

$$ \hat{l}_i^c = p(z_i^{\text{head}}|c) = \exp\left(\frac{-d(z_i^{\text{head}}, \Psi^c)^2}{2\sigma^2}\right), \qquad (5.13) $$

where d is a distance measure over the representation space, in our experiments, d is the Euclidean distance. Note that in our case, the embeddings are normalized after their computation, therefore the Euclidean distance is equivalent to the Cosine Similarity. σ is the standard deviation of the distribution and is set to 0.5 in our experiments. In Faster R-CNN, the background is considered as a class as well, the corresponding score can be derived from the other class scores as follows:

$$ \hat{l}_i^\emptyset = p(z_i^{\text{head}}|\emptyset) = 1 - \max_{c \in \mathcal{C}} \hat{l}_i^c, \qquad (5.14) $$

where $\emptyset$ denotes the background class.

In the RPN the objectness computation is very similar to the classification score in the head. However, only two classes are considered: foreground and background. The foreground class is seen as a mixture of Gaussians (i.e., a mixture of all foreground classes) and is approximated as the maximum score among all classes for stability reasons:

$$ \hat{o}_i = \max_{c \in \mathcal{C}} \hat{l}_i^c. \qquad (5.15) $$

These modifications make Faster R-CNN able to adapt to novel classes. Computing prototypes for novel classes allows direct adaptation of the whole detection model and not simply the detection head as in RepMet. However, with these changes, the model also requires a different training scheme to ensure that the prototypes are properly leveraged and classes are not only memorized.

### 5.2.2 Training Procedure

Before presenting the changes with the Faster R-CNN training procedure, we present here what remains unchanged: the loss functions and the example selection. Faster R-CNN is trained using four distinct loss functions, two for the RPN and two for the detection head:

$$ \mathcal{L}_{reg}^{\text{RPN}}(b_i^{\text{RPN}}, \bar{b}_i^{\text{RPN}}) = \text{SmoothL1Loss}(b_i^{\text{RPN}}, \bar{b}_i^{\text{RPN}}), \qquad (5.16) $$

$$ \mathcal{L}_{obj}^{\text{RPN}}(o_i, \hat{o}_i) = \hat{o}_i \log(o_i) + (1 - \hat{o}_i) \log(1 - o_i), \qquad (5.17) $$

$$ \mathcal{L}_{reg}^{\text{head}}(b_i^{\text{head}}, \hat{b}_i^{\text{head}}) = \text{SmoothL1Loss}(b_j^{\text{head}}, \hat{b}_j^{\text{head}}), \qquad (5.18) $$

$$ \mathcal{L}_{cls}^{\text{head}}(c_i, \hat{c}_i) = -\log(l_i^c), \qquad (5.19) $$
where $b_i^{\text{RPN}}$ and $o_i$ are the ground truth targets for the regression and classification branches of the RPN. Similarly, $b_i^{\text{head}}$ and $c_i$ are the target for the detection head. During training, not all boxes are selected for computing the losses. The generated boxes (or proposals for the RPN) are separated into two groups: positive examples, i.e. boxes with an overlap of at least 0.7 with a ground truth annotation, and negative examples which represent the background class. The classification losses are computed over all examples, while the regression losses only take into account the positive boxes. This remains unchanged for Prototypical Faster R-CNN.

However, the training of PFRCNN is done episodically, following the Meta-Learning paradigm and the training scheme proposed in Prototypical Networks [126]. The motivation behind such training is to mimic the setup that will be encountered at test time and prevent base classes memorization. Indeed, during base training, only annotations from the base classes are available. Training the model with all base classes at the same time could lead to overfit the base class set, at the cost of adaptation. The episodic training consists in sampling a subset of classes $C_{\text{ep}} \subset C_{\text{base}}$ and train the model to detect only these classes for a few training steps. Such a training phase is called an episode. The episodes are then repeated over and over until convergence. During each episode, a *query set* and a *support set* are sampled from the original dataset. The support set contains the examples that will be leveraged for the prototypes computation. On the other hand, the query set is exploited as a small training set. The loss is computed on the query set and between each update of the model, the prototypes are re-computed from the same support set. The update of the prototype is not necessary between each training step, but since the model's weights are updated, the class representations also change. Additionally, the episodic strategy allows for mimicking the test time setting. If there are *N* novel classes with *K* support images at test time, the episodes can reproduce this even though the dataset has a lot more classes and data. Episode after episode, the model will encounter new class combinations and support examples, in the end, it should learn to generalize to novel classes from a few examples, according to Meta-Learning claims.

To build the support set, for each class $c \in C_{\text{ep}}$, we select images containing objects of class *c* and disregard all other objects (i.e. their annotations are not included in the support set but the image is not masked, so they are still visible). If there is more than one object *c* in the image, only one is selected randomly as the annotated example. This prevents having more than *K* examples per class. The query set contains $K_{\text{query}}$ images for each of its *N* classes, this means at least $K_{\text{query}}$ examples for each class, but this number can be larger as more than one object is present in the images. As for the support set, the annotations with class labels not in $C_{\text{ep}}$ are discarded. This sampling procedure prevents the occurrence of *self-distractors* but not co-occurrence distractors (see Sec. 4.1).

Once the base training is done, the network can directly be applied to novel classes through direct adaptation from the prototypes (see Fig. 3.4). However, the adaptation is only performed in the classification parts of the model, regression branches are not modified. This is certainly sub-optimal and therefore, we provide a fine-tuning scheme to remedy this. This fine-tuning is done exactly as the base training phase, in an episodic manner except that the episode classes are sampled from
both base and novel class sets: $C_{ep} \subset (C_{base} \cup C_{novel})$. In this case, the examples of novel classes are the same in the query and support sets so that the total number of support examples remains fixed.

### 5.2.3 Iterative improvements

PFRCNN, as described in the previous section, denoted the baseline, does not perform well on aerial images (see Tab. 5.1). Therefore, we introduce a series of improvements to improve the performance of the model.

In this section, we propose a series of improvements on top of the PFRCNN baseline described above. Indeed, when tested on aerial images, vanilla PFRCNN yields relatively poor performance (see *baseline* performance in Tab. 5.1). To remedy this, we introduce several training tricks.

#### Hard negative example mining

One issue encountered with the baseline is the detection of the base classes regardless of support examples. Basically, it detects base classes even though no prototypes are provided for these classes: this is base class memorization. Although this improves performance when base class prototypes are provided, it produces lots of false positive detections when novel classes are wanted. To address this, we propose to sample hard negative examples to encourage the model to detect support classes only. The main idea is to take advantage of the annotations for classes not selected in the current task to find hard negative examples, i.e. classes that the network could have memorized from previous tasks but should not be detected during this episode. When starting a new episode, it is likely that the model still produces detection for objects annotated in one of the previous episodes if it does not rely on the support information. Even though these objects are not annotated in the new task, their annotations are available in the dataset (because they belong to base classes). Therefore, these annotations can be used to find examples that should be considered as background for the current task. They are different from the background examples that do not contain any class of the dataset, which are referred to as easy negative examples and are much more numerous. Explicitly sampling these hard negative examples encourages the network to detect only objects annotated in the support set.

#### Moving average prototypes

Another issue with the baseline is that the prototypes can change abruptly, either when the network is updated or when the support set changes. We argue that this causes some training instabilities. To prevent such rapid modification of the prototypes, an exponential moving average is introduced to smooth the disruption. Hence, $\bar{\Phi}_{t+1}^c = \alpha\Phi_t^c + (1-\alpha)\bar{\Phi}_t^c$. $\alpha$ is set to 0.1 in our experiments. $\bar{\Phi}_t^c$ is the averaged prototype for class c at iteration t, while $\Phi_t^c$ is the prototype computed from the support set, for class c at iteration t.

#### Background clustering

Lastly, the baseline shows a poor separation of novel class representations (see Fig. 5.5). This leads to poor performance with novel classes at test time. In order to solve this, inspiration is drawn from
[295]. At each iteration, they fit a K-means on the learned representations. This gives pseudo-labels to train the network for classification in a self-supervised manner. Similarly, we propose to fit a K-means on the negative embeddings (i.e. representing boxes not matched by any ground truth object). From the resulting pseudo-labels a contrastive loss function (Triplet Loss [296]) is computed. The triplets are formed with embeddings labeled identically by the K-means. It encourages the network to organize the negative examples into tight and separated clusters. This will eventually discover semantic clusters that represent novel objects.

## Ablation study

In order to assess the relevance of the tricks formulated in previous paragraphs, a small ablation study is conducted on DOTA dataset. The results of this analysis can be found in Tab. 5.1. On the one hand, the introduction of hard examples mining and moving average prototypes improves consistently the novel classes mAP in the one-shot setting. On the other hand, background clustering greatly reduces the performance on base classes, while achieving similar results on novel classes. According to this analysis, we chose to keep only hard example mining and the moving average as it combines the best base and novel classes performance.

<table><caption>Table 5.1: Ablation study about the training tricks described in section 5.2.3. Each column corresponds to the addition of each trick on top of the previous one. HEM, MA and BC correspond respectively to Hard Example Mining (HEM), Moving Average (MA) prototypes and Background Clustering (BC). Detection performance is reported as mAP with a 0.5 IoU threshold. Blue and red values represent the best performance on base and novel classes respectively.</caption><thead><tr><th rowspan="2">K</th><th colspan="2">PFRCNN Baseline</th><th colspan="2">+HEM</th><th colspan="2">+MA</th><th colspan="2">+BC</th></tr><tr><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th></tr></thead><tbody><tr><td>1</td><td>35.5</td><td>2.1</td><td>31.2</td><td>4</td><td>26.5</td><td>6.9</td><td>13.3</td><td>4.3</td></tr><tr><td>3</td><td>35.9</td><td>2.7</td><td>35.6</td><td>2.3</td><td>33.9</td><td>3.5</td><td>14.5</td><td>4.1</td></tr><tr><td>5</td><td>34.3</td><td>3.8</td><td>41.2</td><td>3.3</td><td>37</td><td>4.2</td><td>18.2</td><td>4.7</td></tr><tr><td>10</td><td>30.4</td><td>4.1</td><td>34.3</td><td>2.6</td><td>35.1</td><td>5.9</td><td>14.8</td><td>2.6</td></tr></tbody></table>

## 5.3 Performance on Artificial Data

Before applying PFRCNN on aerial images, we test it on an artificial dataset with reduced difficulty. This gives a hint about the capacities of the model on real data.

### 5.3.1 MNIST-LOC Dataset

As an artificial dataset, we leveraged MNIST-LOC. This dataset is not a published work but rather a toy example sometimes mentioned in the literature. It consists in creating artificial images with the handwritten digit images from the original MNIST dataset [297]. For each image in MNIST-LOC, a random number of MNIST digits are sampled and placed randomly in the image with a random scale. This creates a potentially infinite dataset but with limited variability. For our experiments, we build a dataset with 20k images in the training split and 2k images both for the test and validation splits.
<figure><img src="image_63.png" /><figcaption>Figure 5.2: Images of MNIST-LOC dataset and classes repartition.</figcaption></figure>

The dataset has around 120k annotated objects, which translates to approximately 12k instances per class. An overview of the dataset is provided in Fig. 5.2.

Compared to a real dataset, MNIST-LOC is far more simple. The background is uniform which simplifies the localization of the objects. Then, the class occurrences are uniformly distributed. And finally, the intra-class variance is reduced as MNIST is an easy dataset for the classification task.

### 5.3.2 Implementation details

We provide here some of the implementation details for training Prototypical Faster R-CNN, but a complete list of the hyperparameters and their values is available in our GitHub<fnref n="1" />. The optimization is done with Adam optimizer [298] and a learning rate of 1e-4. The backbone network is pre-trained on ImageNet and its first two layers are kept frozen during training. Three classes are selected as novel classes and are reserved for evaluation, the 7 others are kept as base classes. Each episode is constituted of $K_{query}$ = 5 images per class, i.e., 15 images per episode.

### 5.3.3 Detection performance on MNIST-LOC

We present the performance results on MNIST-LOC in Tab. 5.2. This table reports the mean Average Precision (mAP) with an IoU threshold of 0.5 (see Sec. 2.1.2 for more details about mAP). The results are given with multiple values of *K*, the number of support examples, and two distinct splits of the base and novel classes. The evaluation is done on an unseen test set, from which the support examples are sampled as well. The table provides the mAP both for base and novel classes separately as we do not consider the generalized few-shot setting.

<footnote n="1">¹Link to Prototypical Faster R-CNN repository.</footnote>
<table><caption>Table 5.2: PFRCNN performance on MNIST-LOC dataset with two distinct class splits. On the left, novel classes are 0, 1 and 4, while on the right novel classes are 3, 5, and 8. In both cases, all other classes belong to the base class set. Performance is reported as mAP<sub>0.5</sub>. The last row reports the performance of a vanilla Faster R-CNN trained in a regular data regime, i.e., with all available annotations in the dataset. For Faster R-CNN, per-class performance is averaged over base and novel classes separately to compare with the few-shot techniques.</caption><thead><tr><th rowspan="2">Data regime</th><th colspan="2">Split 1: [0, 1, 4]</th><th colspan="2">Split 2: [3, 5, 8]</th></tr><tr><th>Base Classes</th><th>Novel Classes</th><th>Base Classes</th><th>Novel Classes</th></tr></thead><tbody><tr><td>1 shot</td><td>94.86</td><td>21.86</td><td>92.46</td><td>19.43</td></tr><tr><td>3 shots</td><td>95.70</td><td>20.39</td><td>94.82</td><td>21.22</td></tr><tr><td>5 shots</td><td>95.10</td><td>24.34</td><td>94.95</td><td>21.73</td></tr><tr><td>10 shots</td><td>95.86</td><td>23.19</td><td>93.11</td><td>20.17</td></tr><tr><td>Faster R-CNN</td><td>76.86</td><td>96.33</td><td>84.29</td><td>79.01</td></tr></tbody></table>

First, it can be seen from this table that the performance in a regular data regime (i.e., vanilla Faster R-CNN with all annotations) is high. This confirms that MNIST-LOC is a fairly simple dataset and that the detection task is way easier on this dataset than on real ones. It is important to note that these values cannot be directly compared with the performance values in the few-shot regime as the number of classes is different. In the regular regime, the classification problem has 10 classes whereas, in the few-shot regime, it only has three (3-ways K shots setting, even for base classes). Then, the few-shot performance of PFRCNN on base classes is also quite high, approaching one as the number of shots grows. However, for novel classes, this is different, the mAP values are way lower in this case and fall below an acceptable threshold for any industrial use case. To get a better grasp on these results, Fig. 5.3 gives detection examples on MNIST-LOC dataset for base and novel classes. For base classes, two distinct support sets are leveraged between rows 1 and 2 (with different classes, i.e., $C_{ep}^1 \neq C_{ep}^2$). For base classes, the detection is almost perfect, which represents well the scores from Tab. 5.2. However, for novel classes, there are undesired detections of base classes and a lot of confusion between novel classes.

## 5.4 Difficulties on Aerial Images

While the Prototypical Faster R-CNN is challenged on synthetic images, it has more serious difficulties with real images. In this section, we present the detection result of PFRCNN applied on aerial images, specifically on DOTA and DIOR datasets.

First, Tab. 5.3 gathers the performance results of PFRCNN on DOTA dataset for base and novel classes. As for MNIST-LOC, two distinct class splits are experimented: Split A with *plane*, *ship* and *tennis-court* and Split B with *harbor*, *roundabout* and *helicopter*. Following the same configuration as in the previous section, we report the performance with mAP<sub>0.5</sub> for base and novel classes independently. The results on base classes are much lower than with MNIST-LOC, but it makes sense as the detection task in DOTA is also much more difficult. Nevertheless, the base classes' performance is
<figure><img src="image_64.png" /><figcaption>Figure 5.3: Prototypical Faster R-CNN qualitative detection results on MNIST-LOC dataset, on base and novel classes. Predictions are done without fine-tuning and with *K* = 1.</figcaption></figure>
<table><caption>Table 5.3: PFRCNN performance on DOTA dataset with two distinct class splits. Split A has classes *plane*, *ship* and *tennis-court* and Split B has *harbor*, *roundabout* and *helicopter*. In both cases, all other classes belong to the base class set. Performance is reported as mAP<sub>0.5</sub>. The last row reports the performance of a vanilla Faster R-CNN trained in a regular data regime, i.e., with all available annotations in the dataset. For Faster R-CNN, per-class performance is averaged over base and novel classes separately to compare with the few-shot techniques.</caption><thead><tr><th rowspan="2">K shots</th><th colspan="2">Split A</th><th colspan="2">Split B</th></tr><tr><th>Base classes</th><th>Novel Classes</th><th>Base classes</th><th>Novel Classes</th></tr></thead><tbody><tr><td>1</td><td>27.5 ± 1.0</td><td>4.7 ± 2.0</td><td>41.5 ± 3.0</td><td>8.0 ± 1.0</td></tr><tr><td>3</td><td>35.2 ± 2.0</td><td>2.4 ± 1.0</td><td>39.2 ± 3.0</td><td>10.1 ± 2.0</td></tr><tr><td>5</td><td>39.0 ± 1.0</td><td>3.8 ± 1.0</td><td>43.4 ± 2.0</td><td>12.1 ± 1.0</td></tr><tr><td>10</td><td>38.4 ± 2.0</td><td>4.1 ± 1.0</td><td>41.4 ± 3.0</td><td>10.1 ± 2.0</td></tr><tr><td>Faster R-CNN</td><td>65.62</td><td>90.96</td><td>73.21</td><td>69.77</td></tr></tbody></table>

much lower than the regular setup (i.e., Faster R-CNN trained on the whole DOTA). For novel classes, a similar performance drop is observed, making PFRCNN unfit for any industrial application.

Nonetheless, these experiments are not useless and provide relevant insights about the FSOD task and its difficulties. For instance, with the MNIST-LOC dataset, almost no difference could be seen between splits. With DOTA, much better performance is achieved on Split B than Split A. It indicates some interactions between classes, some combinations are more difficult than others. These considerations were not taken into account in the design of PFRCNN and should be overcome to achieve reasonable few-shot detection.

Despite its limited performance, Prototypical Faster R-CNN is one of the first approaches to tackle FSOD on remote sensing images from a metric learning perspective. In addition, this method does not need any fine-tuning. All previous results were given from a simple adaptation to the novel classes at inference time with novel prototypes. We also experimented with an additional fine-tuning step, especially to refine the regression branches of the model. This was performed on DOTA with Split A and the results are available in Tab. 5.4. Fine-tuning with the few available support examples helps significantly to boost the detection quality on novel classes, but it remains insufficient for COSE's application. Interestingly, after fine-tuning a common property of few-shot methods emerges: the more examples are provided, the higher the performance. It was not the case without fine-tuning. With Split A, the best performance is achieved with *K* = 1, with Split B, it increases until *K* = 5 and then decreases with *K* = 10. This indicates that the management of more shots is difficult within PFRCNN. It suggests that support examples features may not be trivially aggregated as it can produce irrelevant prototypes. This can happen when a class has a great variety and thus a multimodal distribution in the embedding space.

Just as for MNIST-LOC, we provide qualitative results of the FSOD on DOTA with PFRCNN. These are available in Fig. 5.4. The detection is satisfactory (but not perfect) on the base classes. However, the bounding boxes and labels for novel classes (bottom 2 rows) are mostly incorrect. Some
<table><caption>Table 5.4: PFRCNN performance comparison with and without fine-tuning.</caption><thead><tr><th rowspan="2">K shots</th><th colspan="2">Without fine-tuning</th><th>With fine-tuning</th></tr><tr><th>Base classes</th><th>Novel Classes</th><th>Novel Classes</th></tr></thead><tbody><tr><td>1</td><td>27.5</td><td>4.7</td><td><strong>7.5</strong></td></tr><tr><td>3</td><td>35.2</td><td>2.4</td><td><strong>9.3</strong></td></tr><tr><td>5</td><td>39.0</td><td>3.8</td><td><strong>11.3</strong></td></tr><tr><td>10</td><td>38.4</td><td>4.1</td><td><strong>11.6</strong></td></tr></tbody></table>

<figure><img src="image_65.png" /><figcaption>Figure 5.4: Prototypical Faster R-CNN qualitative detection results on DOTA dataset, on base and novel classes. Predictions are done without fine-tuning and with *K* = 1.</figcaption></figure>
<figure><img src="image_66.png" /><figcaption>Figure 5.5: TSNE visualization of the trained embedding space of PFRCNN on DOTA. Each point represents the projection of RoI in the embedding space. Large circles and squares respectively denote the prototypes of base and novel classes. Black points denote background proposals.</figcaption></figure>

confusion between base and novel classes occurs. For instance, in the left-most image in the third row, water tanks are mistaken as roundabouts. Of course, these two classes look similar in practice and that makes them difficult to distinguish. To better understand why this confusion happens, we investigate the embedding space of PFRCNN through TSNE visualization (see Fig. 5.5). This figure is made by collecting the embedding vectors of all proposals over an entire query set, and then by reducing their dimension using the TSNE algorithm [299]. Class-specific clusters are well-formed in the representation space, but some classes overlap which explains the confusion. Representations of these classes may be close to another class prototype and get misclassified. This is especially true for novel classes which overlap over base classes, explaining their poor performance. For the example above, the misclassification of the two water tanks is easily understood from the TSNE plot as these two classes almost perfectly overlap (class *storage-tank* in dark green and *roundabout* in pink).

## 5.5 Insights and conclusion

From the results presented above, one question arises: is representation learning a suitable choice for few-shot object detection? Metric-learning methods are competitive with state-of-the-art for few-shot classification but seem less appropriate for FSOD. Prototypical Faster R-CNN is a first attempt to apply prototypical networks to FSOD. The very few FSOD approaches based on Metric-
Learning often leverage other tricks such as carefully designed fine-tuning or attention mechanisms. Furthermore, at the beginning of this PhD, there were only two contributions solving FSOD with metric learning: RepMet [251] and RN-FSOD [252]. More investigation was therefore needed in this direction. Of course, the poor results of PFRCNN alone are not enough to conclude that all metric-learning-based approaches are inappropriate. Nevertheless, metric-learning FSOD methods are now in the minority in the current literature, which indirectly confirms their inadequacy.

Despite the relatively poor performance of PFRCNN, our experiments provide useful insights for future designs of FSOD methods. First fine-tuning is crucial for FSOD. It yields significant performance gains compared to models only trained on base classes. This makes sense as the adaptation of the model with the prototypes is only performed in the classification branches. The regression branches are therefore unprepared for the localization of novel classes. Of course, having a method that does not require any fine-tuning is highly desirable from an industrial perspective, but that should not come at the cost of poor performance. Then, Faster R-CNN may not be the best detector choice for few-shot extensions. Indeed, its two-stage structure duplicates the number of modifications required for the adaptation to novel classes. Even if some works argue that the RPN is class-agnostic, it is still trained to only detect base classes while discarding everything else, including potential novel classes. The RPN must then be adapted to novel classes as well. It makes the few-shot extension more cumbersome, with more parameters and more causes for failure. One-stage detectors certainly are a more sensible choice. Finally, the episodic training strategy may also be inadequate for detection. It complexifies greatly the training and introduces distractors (this concept is explained in Sec. 4.1). At the beginning of each episode, a subset of classes (either base or novel depending on the training phase) is sampled. Annotations from all other classes are discarded during the episode, yet the training images still contain instances of other classes. These distractors are confusing for the model. Of course, for classification, the episodic strategy forces the model to establish connections between support examples and the query images. But it is much simpler as the query images only contain one object belonging to one of the episode classes. The presence of already-seen classes (and potential future classes) inside query images certainly makes the episodic training strategy suboptimal for the detection task.

The latter paragraph formulates a few assumptions based on our observations of the design and training of PFRCNN. Of course, it would be wise to conduct dedicated experiments to confirm these hypotheses. For instance, carefully designed synthetic images could help to experiment with the distractors' influence in a controlled way.

To conclude, this chapter presents Prototypical Faster R-CNN, a fully metric-learning-based approach for the Few-Shot Object Detection task. This is one of the first methods proposed in this category. Despite its relatively poor performance on real images, it can adapt to novel classes without any fine-tuning, which is still a rare property in the current literature. Finally, the experiments conducted with PFRCNN provide relevant insights about the FSOD task and will help in the design of future approaches.
CHAPTER 6

ATTENTION FRAMEWORK FOR FAIR FSOD
COMPARISON

ABSTRACT

Fair comparison is extremely challenging in the Few-Shot Object Detection task as plenty of
architectural choices differ from one method to another. Attention-based approaches are no
exception, and it is difficult to assess which mechanisms are the most efficient for FSOD. In
this chapter, we propose a highly modular framework to implement existing techniques and
design new ones. It allows for fixing all hyperparameters except for the choice of the attention
mechanism. Hence, a fair comparison between various mechanisms can be made. Using
the framework, we also propose a novel attention mechanism specifically designed for small
objects.

<figure><img src="image_67.png" /></figure>

P. Le Jeune and A. Mokraoui, "A Comparative Attention Framework for Better Few-Shot Object Detection on Aerial Images", Submitted at the Elsevier Pattern Recognition journal.

<figure><img src="image_68.png" /></figure>

P. Le Jeune and A. Mokraoui, "Cross-Scale Query-Support Alignment Approach for Small Object Detection in the Few-Shot Regime", Accepted at the IEEE International Conference on Image Processing 2023 (ICIP).

CONTENTS

<table>
  <tbody>
    <tr>
      <td>6.1</td>
      <td>Framework Presentation and Motivation</td>
      <td>106</td>
    </tr>
    <tr>
      <td>6.2</td>
      <td>Alignment Attention Fusion Framework</td>
      <td>107</td>
    </tr>
    <tr>
      <td>6.2.1</td>
      <td>Query-Support Alignment</td>
      <td>107</td>
    </tr>
    <tr>
      <td>6.2.2</td>
      <td>Global Attention</td>
      <td>109</td>
    </tr>
    <tr>
      <td>6.2.3</td>
      <td>Fusion Layer</td>
      <td>109</td>
    </tr>
    <tr>
      <td>6.2.4</td>
      <td>Implementation details</td>
      <td>110</td>
    </tr>
  </tbody>
</table>
<table><tr><td>6.2.5</td><td>Fair comparison of Few-Shot Object Detection Methods with AAF.</td><td>115</td></tr><tr><td>6.3</td><td>Cross-Scales Query-Support Alignment for Small FSOD.</td><td>120</td></tr><tr><td>6.3.1</td><td>XQSA definition.</td><td>120</td></tr><tr><td>6.3.2</td><td>Ablation study XQSA.</td><td>122</td></tr><tr><td>6.3.3</td><td>Application to aerial and natural images.</td><td>123</td></tr><tr><td>6.3.4</td><td>XQSA Implementation Challenges and Extensions</td><td>126</td></tr><tr><td>6.4</td><td>Qualitative Comparison within the AAF framework.</td><td>127</td></tr><tr><td>6.4.1</td><td>Base Classes Detection Quality.</td><td>127</td></tr><tr><td>6.4.2</td><td>Novel Classes Detection Quality.</td><td>127</td></tr><tr><td>6.5</td><td>Conclusion.</td><td>129</td></tr></table>

## 6.1 Framework Presentation and Motivation

As seen in Chap. 5, metric-learning approaches are not the optimal choice for the FSOD task. The early FSOD literature has been dominated by attention-based methods, which probably are a more sensible alternative. Plenty of contributions in this domain leverage attention mechanisms for solving the detection task. However, it is difficult to make fair comparisons between the various mechanisms. Each method is proposed with its own choice of detection framework, backbone, hyper-parameters, loss function, augmentations and training strategy. Thus, it is difficult to demonstrate the superiority of one attention mechanism over others. Furthermore, there is no consensus in the FSOD field about a proper way to evaluate the models. This can change from one work to another and is also a source of variance preventing fair comparison in the literature. To this end, we propose a modular framework called the Alignment-Attention-Fusion (AAF) framework. The goal of this framework is to allow the implementation of various attention mechanisms while keeping all other parameters fixed. Looking closely at the existing attention-based method in the literature (see Sec. 3.2.3), three main types of attention mechanisms can be observed: Spatial Alignment, Global Attention and Direct Feature Fusion. Therefore the AAF framework is structured around these three components. The framework proposes first a mathematical formalism to present and define existing and future mechanisms. Second, a modular Python package<fnref n="1" /> allows easy implementation of attention-based methods inside a controlled detection environment to ensure fair comparisons. In the following sections, we will present the framework in detail and conduct fair comparison experiments with it. Finally, a novel attention mechanism will be presented, it is designed through the AAF framework and specifically tackles the small objects to improve the detection performance on aerial images.

<footnote n="1">¹https://github.com/pierlj/aaf_framework</footnote>
<figure><img src="image_69.png" /><figcaption>Figure 6.1: Attention-based Few-Shot Object Detection principle. Query and support images are processed by the backbone before being fed to the query-support combination block. Detection is then performed independently on each class.</figcaption></figure>

## 6.2 Alignment Attention Fusion Framework

In Sec. 3.2.3, three main components of attention mechanisms for FSOD have been identified: Spatial Alignment, Global Attention and Direct Feature Fusion. Most attention-based FSOD methods rely on one or more of these components. This section will cover the Alignment, Attention and Fusion (AAF) framework, whose purpose is to provide a flexible environment to implement attention-based FSOD methods. Before jumping into the definition of the AAF framework, let's recall briefly the main principle of attention-based FSOD, illustrated by Fig. 6.1. The goal of the attention module is to combine the information from the query image and the support examples. Specifically, the query features are compared with class-specific features computed from the support set. This comparison highlights similar parts in the query image and the support examples, yielding class-specific query features. The detection is then performed separately for each class.

The AAF framework takes as input the features from the query image $F_q$ as well as the features extracted from every support image $F_s^c$ for $c \in C$ (if more than one example is available per class $-K > 1$ -, their features are averaged). It outputs class-specific query features $M_q^c$ in which features relative to class $c$ are highlighted. The framework is divided into three parts as shown in Fig. 6.2, which provides an overview of the framework. Each component is described below in dedicated sections with examples of possible design choices. Even though this framework is presented from the perspective of object detection, it could be applied to any kind of few-shot visual tasks (e.g., classification or segmentation).

### 6.2.1 Query-Support Alignment

The alignment module, denoted $\Lambda$, spatially aligns the features from the query and the support. It is unlikely that objects of the same class appear at the same position inside query and support
<figure><img src="image_70.png" /><figcaption>Figure 6.2: The Alignment Attention Fusion (AAF) framework is composed of three components: spatial alignment $\Lambda$, global attention $\Gamma$ and a fusion layer $\Omega$. Examples for each module are depicted, these come from FSOD methods in the literature. Ex. A is presented in [222], Ex. B in [220] and Ex. C in [233]. The colors chosen in this diagram match the colors used in Eqs. (6.1) to (6.5).</figcaption></figure>

images (in comparison to the classification task where objects to classify are in the center of the image). This issue is commonly avoided by pooling the support map and using it as a class-specific reweighting vector. But as discussed in Sec. 3.2.3, this trick loses the spatial information about the support object, which can be detrimental for detection. Instead, an alignment based on similarity can be done between query and support feature maps. The idea is to re-organize one feature map by comparing it with the other so that similar features are spatially close in the maps (see Fig. 3.3). The alignment module is defined as follows:

$$A_q^c = \lambda_q(F_q, F_s^c) F_q, \qquad (6.1)$$

$$A_s^c = \lambda_s(F_q, F_s^c) F_s^c. \qquad (6.2)$$

<figure><img src="image_71.png" /></figure>

The definition of the matrices $\lambda_q$ and $\lambda_s$ determines how features are aligned. They are mostly derived from a similarity measure between query and support features. This formulation is close to the non-local blocks described in [300] and is at the heart of visual transformers [56]. Transformers attention can be understood as an alignment of the value to match the query-key similarity. This formulation allows easy implementation of different feature alignments by changing the expression of the affinity matrices. As an example, Meta Faster R-CNN [222] leverages an alignment module with affinity matrices $\lambda_s(F_q, F_s^c) = F_q \cdot (F_s^q)^T$ and $\lambda_q(F_q, F_s^c) = I$ (see Example A in Fig. 6.2). Only the support features are aligned so that they match query features. It is important to mention that alignment alone does not combine query and support features. It rather reorganizes spatially the query or support features. However, once the features are aligned, their sizes match, which allows direct comparison through element-wise operations (within the fusion layer).
## 6.2.2 Global Attention

The global attention module, denoted $\Gamma$, combines global information of the supports and the query. It highlights class-specific features and softens irrelevant information for the task. This module is defined as follows:

$$H_q^c = \gamma_q(A_q^c, A_s^c), \qquad (6.3)$$

$$H_s^c = \gamma_s(A_q^c, A_s^c). \qquad (6.4)$$

Merged query features

Fusion operator

The global attention operators $\gamma_q$ and $\gamma_s$ combine the global information from their inputs and highlight features accordingly. This is generally done through channel-wise multiplication. In this way, class-specific features are highlighted, while features not relevant to the class are softened. Changing the definition of $\gamma_q$ and $\gamma_s$ allows the implementation of a wide variety of global attention mechanisms. This technique largely resembles the principle of the Learnet [144] introduced for FSC. For instance, reference [220] pools the support maps with a global max pooling operation (GP) into a reweighting vector and reweights the query features through channel-wise multiplication: $\gamma_q(A_q^c, A_s^c) = A_q^c \otimes GP(A_s^c)$ and $\gamma_s(A_q^c, A_s^c) = A_s^c$ (see Example B in Fig. 6.2).

## 6.2.3 Fusion Layer

The purpose of the fusion component is to combine highlighted query and support maps. This is only applicable when the maps have the same spatial dimension. It is mostly used alongside the alignment module as it does not combine the information from the support and the query but only reorganizes the maps. In particular, when support and query maps do not have the same spatial dimension, aligning support maps with query maps fixes the size discrepancy. The fusion module is then defined as follows:

$$M_q^c = \Omega(H_q^c, H_s^c). \qquad (6.5)$$

Merged query features

Fusion operator

The highlighted maps can be combined through any point-wise operation, addition $\oplus$, multiplication $\odot$, subtraction $\ominus$, concatenation $[\cdot, \cdot]$ or more sophisticated ones. An example of such a fusion module is presented in [233]. The fusion operator concatenates the results of the addition and the subtraction of the highlighted features: $\Omega(H_q^c, H_s^c) = [H_q^c \oplus H_s^c, H_q^c \ominus H_s^c]$ (see Example C in Fig. 6.2). The point-wise operators can also contain small trainable models such as in [222], where small CNNs (e.g., $\psi_{dot}$, $\psi_{sub}$, and $\psi_{cat}$) are applied after the point-wise operators, but before the concatenation: $\Omega(H_q^c, H_s^c) = [\psi_{dot}(H_q^c \odot H_s^c), \psi_{sub}(H_q^c \ominus H_s^c), \psi_{cat}([H_q^c, H_s^c])]$.

Except for the fusion layer which must be applied last, spatial alignment and global attention can be applied in any order. This flexibility is required to cover methods that apply global attention before alignment, such as DANA [234]. The whole architecture of the AAF framework is illustrated in Fig. 6.2, in which examples from the previous sections are also depicted. The modular structure of
the framework enables the implementation of a wide variety of attention mechanisms while keeping
all other hyperparameters fixed. In this way, it is a useful tool for FSOD methods comparison.

**6.2.4 Implementation details**

Before presenting the results of a fair comparison between several FSOD approaches re-implemented
in the AAF framework, we must review its implementation. To make the comparisons fair, some
design choices are kept fixed in the framework. The backbone is a ResNet-50 with a 3-layers Feature
Pyramid Network on top. It extracts features at 3 different levels, which helps the network to detect
objects of various sizes. The detection head is based on FCOS [45], a one-stage detector (a choice
motivated by insights from Chap. 5). The head is composed of a few convolutional layers with
ReLU activations followed by two branches (convolutional as well) for classification and regression
respectively. The AAF framework is applied between the backbone and the detection head. As
features are extracted at multiple levels, attention mechanisms are also implemented to work at
different scales. This may differ from the original implementations, but most methods are already
designed to work at multiscale (see Tab. 3.1). The model is trained in an episodic manner. During
each episode, a subset $C_{ep} \subset C$ of the classes is randomly sampled. A query set is sampled for
each episode, containing 100 images per class. This set only contains annotations of the episode
classes and is leveraged for the loss computation and the optimization of the model. A support
set is also sampled at the beginning of each episode containing $K$ examples for each episode class.
The support examples are used through the attention mechanisms to condition the detection on the
episode classes.

The training is divided into two phases *base training* and *fine-tuning*. During base training, only base classes can be sampled ($C_{ep} \subset C_{base}$) and one image per class is drawn for the support set ($K = 1$). The optimization is done with SGD and a learning rate of $1 \times 10^{-3}$ for 1000 episodes. During *fine-tuning*, the backbone is frozen, the learning rate is divided by 10, and the episode classes can be sampled from $C_{base} \cup C_{novel}$, with at least one novel class per episode. Examples from novel classes are selected among the $K$ examples sampled once before fine-tuning. Each model is fine-tuned separately for different values of $K \in \{1, 3, 5, 10\}$. During both training phases, the same loss function is optimized, as defined in FCOS (see Tab. 2.1).

**6.2.4.1 Augmentations and Cropping Strategies**

Some existing works leverage sophisticated training strategies (e.g. auxiliary loss functions [224],
hard examples mining [260] or multiscale training [223]). While this certainly improves the quality
of the detections, it introduces new parameters to tune and makes the comparison with other works
difficult. As the focus of this study is on attention mechanisms, we choose not to reimplement
all these improvements. However, to remain competitive with existing works, we propose a novel
augmentation pipeline specifically designed for object detection. It is defined in the next paragraph
which includes a cumulative study of its different components on DOTA. In addition, we discuss
the choice of the support extraction strategy. Basically, this refers to how the support examples are
extracted from the support images since most parts of these images are irrelevant for the task. From
our analysis, it seems that this design choice significantly influences the model performance (see
Tab. 6.2). However, it is barely discussed in the FSOD literature. We find that the best strategy is to
crop the support example and resize it to a fixed-size patch. This strategy is thus fixed for all our
experiments.

Augmentations

To improve the performance of the methods implemented in the AAF framework and be competitive with existing works, we propose an augmentation pipeline specifically designed for detection. Some regular augmentation techniques cannot be directly applied for object detection as they can completely mask objects from the image. This deteriorates the training as the model will not be able to detect hidden objects, but it will be penalized anyway.

First, we apply random horizontal and vertical flips (only for aerial images) and color jitter. As it does not remove entire objects, these can be applied directly to the images. However, some other classical techniques such as random crop-resize and random cut-out cannot be applied directly. Therefore, we developed object-preserving random crop-resize and cut-out. The main idea is to apply these transformations at the object level and not at the image level. This ensures that objects of interest are still visible in the transformed image. For crop-resize, a non-empty subset of the objects in the image is randomly sampled. An overall bounding box is computed around all these objects and the cropped area is randomly drawn between this box and the image borders. Hence, it guarantees the presence of at least one object inside the cropped image. For cut-out, the principle is similar, instead of cutting out a random part of the image, the cut is applied at the object level so that it does not hide out entire objects. Fig. 6.3 compares the two proposed augmentations with their naive implementations.

We performed a cumulative study to assess the benefits of each component of the augmentation
pipeline. To do so, we implemented Feature ReWeighting (FRW) [220], a well-known FSOD tech-
nique, within the AAF framework. FRW is then trained on DOTA dataset. This experiment is
summarized in Tab. 6.1. It shows that the augmentation is beneficial for the performance on novel
classes but detrimental for base classes. Surprisingly, performance drops on base classes with aug-
mentation. More specifically, it seems that image flips are responsible for the performance loss on
base classes (see first and second columns in Tab. 6.1). Base classes performance drops when adding
flips but remains mostly constant when adding other types of augmentations. One crucial difference
between flips and other augmentations is that we choose to apply flips also on support examples.
This choice was made to increase the diversity of the support set during fine-tuning. For novel
classes, only a few images are available as support during fine-tuning, and we want to avoid overfit-
ting these examples. Although other types of augmentations could have been employed for this, we
wanted to avoid disrupting too much of the information in the support. This choice may be the cause
of the performance drop on base classes. To verify this hypothesis, we conduct a few more experi-
<figure><img src="image_72.png" /><figcaption>Figure 6.3: Difference between naive augmentation techniques (left) and our adaptation to object detection (right). The proposed transformations are applied at the object-level to preserve objects integrity.</figcaption></figure>

ments disabling the flip in the support set. With the *default* cropping strategy (see next paragraph), the experiments confirm the hypothesis: no performance drop is observed when supports are not flipped. However, support augmentation certainly interacts with the support cropping strategy, thus we also tried with the *same-size* cropping strategy. Surprisingly, it does not produce similar results, and in this case, flipping support examples is actually beneficial for base classes performance. This suggests a complex interaction between augmentation on the support set and the cropping strategy. The choice made in our experiments may not be optimal in this regard, and a deeper analysis of this interaction should be conducted in future work (e.g., studying the effect of various augmentations on a synthetic dataset to have better control over the images). Finally, the base class performance loss is compensated by clear improvements on novel classes. As this is the main goal of FSOD, we choose to adopt the original augmentation pipeline, including flips in the support set, for all our experiments. Other augmentations are not applied to the support set to prevent disrupting their representations and therefore the conditioning of the model.
<table><caption>Table 6.1: Cumulative study of the proposed augmentation techniques on DOTA using FRW [220]. mAP<sub>0.5</sub> is reported for different number of shots.</caption><thead><tr><th># Shots</th><th></th><th>Baseline</th><th>+ Flip</th><th>+ Color</th><th>+ Cutout</th><th>+ Crop</th></tr></thead><tbody><tr><td rowspan="2">1</td><td>Base</td><td>48.83</td><td>45.80</td><td>45.96</td><td>47.20</td><td>45.68</td></tr><tr><td>Novel</td><td>6.15</td><td>5.25</td><td>6.92</td><td>6.44</td><td>10.03</td></tr><tr><td rowspan="2">3</td><td>Base</td><td>51.06</td><td>47.70</td><td>47.03</td><td>46.10</td><td>45.22</td></tr><tr><td>Novel</td><td>14.41</td><td>18.59</td><td>18.59</td><td>19.74</td><td>21.95</td></tr><tr><td rowspan="2">5</td><td>Base</td><td>52.66</td><td>49.38</td><td>50.09</td><td>50.28</td><td>48.74</td></tr><tr><td>Novel</td><td>19.25</td><td>23.71</td><td>25.08</td><td>25.01</td><td>25.95</td></tr><tr><td rowspan="2">10</td><td>Base</td><td>53.84</td><td>50.80</td><td>50.77</td><td>50.41</td><td>50.27</td></tr><tr><td>Novel</td><td>28.56</td><td>31.23</td><td>28.08</td><td>34.13</td><td>35.95</td></tr></tbody></table>

## Support example cropping strategy

The support information is only located inside the example bounding box. The remaining part of the support image mostly contains irrelevant information concerning the object class. Therefore, extracting features from the whole support images is not necessary. But features contained only inside the object's bounding box might not be sufficient as well. Context can be extremely valuable in certain cases, especially for small objects. For instance, a car and a small boat could easily be mistaken without context. Close surroundings of the objects can help for recognition.

A common strategy for support information extraction is proposed by [220]. They concatenate in the channel direction the support image with the support object's binary mask (rectangular, computed from the bounding box) and pass this to an extractor network. This has two main drawbacks. First, it is necessary to compute features from the entire support image, which is a loss of resources. Second, the same network cannot be used for extracting features in query and support images as it needs an additional input channel to process the mask. Hence, the network cannot be pretrained beforehand. This design choice is rarely discussed, if ever mentioned, in the literature.

In this section, we explore this design choice by implementing several extraction strategies. We did not reimplement the technique from [220] as it requires to have two different networks for support and query feature extraction. However, some of our techniques are quite close to what they proposed. These techniques are described below and Fig. 6.4 illustrates most of them:

- **Default**: the most naive extraction technique. It consists in cropping the support image around the support object at a fixed size (e.g. 128 × 128). Objects larger than this are simply resized to fit in the patch.

- **Context-padding**: the cropping occurs exactly as with the default strategy, but areas around the objects are masked out. This is close to what was proposed by [220].

- **Reflection**: context is replaced by reflection of the object. In the case of small objects, the support patch can easily be dominated either by irrelevant information or by zeros when using the latter two extraction methods.
<figure><img src="image_73.png" /><figcaption>Figure 6.4: Illustration of the different cropping strategies tested. The mixed strategy is not illus-
trated as it is a combination of *default* and *same-size*.</figcaption></figure>

- **Same-size**: all objects are resized to fill the support patch entirely (preserving the aspect ratio). It does not change the process for large objects, but it prevents smaller objects from being dominated by irrelevant information.

- **Multi-scale**: the object is resized and cropped at 3 different scales. Each scale is responsible for matching small, medium and large objects in query images.

- **Mixed**: it is a combination of the default strategy and *same-size*. Small objects (i.e. $\sqrt{wh} < 32$) are extracted using the default strategy. Larger objects ($\sqrt{wh} \ge 32$) are resized into a patch of 128 x 128 pixels. Therefore, small objects are not upscaled more than 4 times, as they are using the *resize* strategy.

These strategies are compared in Table 6.2. Even though *same-size* gets the best overall results on
novel classes (regardless of object sizes), there is no clear conclusion. It is outperformed by both
*reflection* and *mixed* for base classes. No method outperforms the others on all object sizes, not
even the ones designed to be more robust to size (*multiscale* and *mixed*). The latter two techniques
introduce discrepancies in the features: objects of similar size (e.g., from both sides of the size limit)
are processed differently, resulting in really different features. As *same-size* gives the best results
on novel classes, we choose to use this strategy for all our experiments. Yet, in the light of our
performance analysis in Chap. 4, we can understand some results from Tab. 6.2. The *multiscale*
<table><caption>Table 6.2: Comparison of support extraction strategies on base and novel classes with DOTA dataset and FRW method with 10 shots. mAP<sub>0.5</sub> is reported on all objects and separately on objects of different sizes: small (S), medium (M) and large (L).</caption><thead><tr><th rowspan="2">Cropping Strategy</th><th colspan="4">Base classes</th><th colspan="4">Novel classes</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td>Default</td><td>49.80</td><td>24.26</td><td>57.36</td><td>63.90</td><td>24.97</td><td>7.66</td><td>24.31</td><td>34.48</td></tr><tr><td>Context padding</td><td>50.03</td><td>24.96</td><td>59.72</td><td>63.03</td><td>29.60</td><td>10.71</td><td>26.76</td><td>50.91</td></tr><tr><td>Same-size</td><td>50.63</td><td>30.83</td><td>59.44</td><td>62.47</td><td>32.19</td><td>8.32</td><td>33.21</td><td>56.54</td></tr><tr><td>Multiscale</td><td>51.44</td><td>29.03</td><td>59.78</td><td>63.27</td><td>26.95</td><td>8.44</td><td>33.48</td><td>45.63</td></tr><tr><td>Reflection</td><td>50.28</td><td>26.05</td><td>59.36</td><td>62.49</td><td>25.50</td><td>7.22</td><td>20.66</td><td>44.29</td></tr><tr><td>Mixed</td><td>50.95</td><td>27.16</td><td>60.48</td><td>60.67</td><td>27.96</td><td>9.51</td><td>26.52</td><td>48.89</td></tr></tbody></table>

strategy does not perform very well as it introduces small objects features which seems detrimental for the good conditioning of the network. On the contrary, *same-size* only generates large objects as support which is a better strategy. Finally, *reflection* performs surprisingly well for small objects while preserving their small size. The redundancy generated by the reflection of such small objects certainly reinforces the object's features.

### 6.2.5 Fair comparison of Few-Shot Object Detection Methods with AAF

To showcase the flexibility of the proposed AAF framework, we reimplement and compare multiple existing works. Some methods described in Chap. 3 are selected: FRW [220], WSAAN [229], DANA [234], Meta R-CNN [222] and DRL [233] (see Tab. 3.1). They have been chosen because they represent well the variety of attention mechanisms available in the literature and according to their popularity. FRW is based on class-specific reweighting vectors, WSAAN has a more sophisticated global attention and computes reweighting vectors inside a graph structure. DANA and Meta R-CNN leverage query-support alignment in different manners and DRL only uses a sophisticated fusion layer. Each of these methods has been reimplemented within the AAF framework. Of course, some details differ from the original implementations, but the purpose of this comparison is to compare only the query-support combination module. In particular, the backbone and the training strategy (losses and episode tasks) may differ. We first conduct such a comparative experiment on Pascal VOC [5] and MS COCO [6] datasets. On these datasets, the performance achieved by our implementations is close (i.e., within 2 points of mAP<sub>0.5</sub>) to the values reported in the original papers. Then, we use the framework to compare the performance of some methods on two aerial datasets: DOTA [77] and DIOR [95].

#### 6.2.5.1 Evaluation protocol

The evaluation is also conducted in an episodic manner, following recommendations from [217]. The performance is averaged over multiple episodes, each containing 500 examples for each class and this operation is repeated multiple times with randomly sampled support sets. The query and support examples are drawn from the test set, thus the support examples are different from the ones used during fine-tuning. This prevents overestimations of the performance due to overfitting on
<table><caption>Table 6.3: Performance comparison between five selected methods on Pascal VOC. All are reimplemented with the proposed AAF framework. Mean average precision is reported for each method on base and novel classes separately and for various numbers of shots ($K \in \{1, 3, 5, 10\}$).</caption><thead><tr><th rowspan="2">K</th><th colspan="2">FRW [220]</th><th colspan="2">WSAAN [229]</th><th colspan="2">DANA [234]</th><th colspan="2">Meta R-CNN [222]</th><th colspan="2">DRL [233]</th><th colspan="2">FCOS Baseline</th></tr><tr><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th></tr></thead><tbody><tr><td>1</td><td>59.92</td><td>28.22</td><td>61.70</td><td>30.95</td><td>62.58</td><td><span style="color:red;">32.82</span></td><td>57.85</td><td>30.16</td><td><span style="color:blue;">64.18</span></td><td>27.05</td><td></td><td></td></tr><tr><td>3</td><td>63.33</td><td>31.12</td><td>63.52</td><td><span style="color:red;">42.19</span></td><td><span style="color:blue;">64.18</span></td><td>33.95</td><td>58.70</td><td>36.79</td><td>61.74</td><td>29.64</td><td></td><td></td></tr><tr><td>5</td><td>64.35</td><td><span style="color:red;">46.33</span></td><td>64.68</td><td>46.16</td><td>65.20</td><td>42.58</td><td>62.14</td><td>40.75</td><td><span style="color:blue;">66.45</span></td><td>37.34</td><td>65.47</td><td>68.02</td></tr><tr><td>10</td><td>63.16</td><td>48.71</td><td>65.27</td><td><span style="color:red;">51.70</span></td><td>65.03</td><td>50.30</td><td>63.38</td><td>49.45</td><td><span style="color:blue;">66.98</span></td><td>47.99</td><td></td><td></td></tr></tbody></table>

the support examples. To ensure a fair comparison between the various methods, the same random seed is used for all evaluations, thus the support and query examples are the same.

### 6.2.5.2 Natural images

Tab. 6.3 gather the results on Pascal VOC. First, compared to the FCOS baseline, a slight performance drop on base classes is observed. This is expected, even if the model has seen a lot of examples of these classes during training, its predictions are still conditioned on a few examples, which can sometimes be misleading. On the other hand, performance on novel classes is significantly lower than the FCOS baseline, especially for low numbers of shots. The number of shots is crucial for performance on novel classes. The higher the number of shots, the better the network performs. On average, with 10 examples per class, the network achieves 0.2 higher mAP$_{0.5}$ than with 1 example. More examples provide more precise and robust class representations, improving the detection. The same phenomenon is observed with base classes to a lesser extent (+0.04 mAP from 1 to 10 shots). Fig. 6.5 displays these trends clearly, both for base and novel classes. In addition, Fig. 6.6 provides the same results split by class. An interesting observation from this last figure is the very good detection performance for the novel class sheep. This can be explained easily from the presence of the class horse in the base set. The model has seen a lot of examples of horses during base training, which makes it learn visual attributes common with a sheep (e.g., four legs, hair and grassy background). Such a class similarity makes the novel class detection much easier. Some authors do leverage this fact, as for instance [264] which first associates a base class to each novel class before learning to discriminate between them.

The behavior just described is expected from any few-shot object detection method. Moreover, performance values are close to what is reported in the original papers of the reimplemented methods. These results are not the same as many architectural choices differ from the proposed methods (e.g. backbone, class splits, etc.). Nevertheless, it confirms that the proposed AAF framework is flexible enough to implement a wide variety of attention mechanisms. It is therefore an appropriate tool to compare and design query-support attention mechanisms.

DRL is arguably the simplest method among the five selected as it leverages only a fusion layer. It combines query features with the features of each support image through concatenation and point-wise operations, creating class-specific query features. It is therefore the closest to the regular
<figure><img src="image_74.png" /><figcaption>Figure 6.5: Evolution of mAP<sub>0.5</sub> with the number of shots averaged on base and novel classes separately. Each line represents one of the reimplemented methods.</figcaption></figure>

FCOS functioning. This explains the very good performance on base classes and lower mAP on novel classes, compared to the baseline. Regarding the other methods, FRW and WSAAN can be easily compared as both are based on global attention. The only difference is how the class-specific vectors are computed. In FRW, they are globally pooled from the support feature map. However, WSAAN combines the same vectors with query features in a graph. This certainly provides better class-specific features and in the end, better results both on base and novel sets. The remaining methods, DANA and Meta R-CNN both leverage spatial alignment. While it seems to bring quite an improvement for DANA over FRW and DRL, the gain is smaller for Meta R-CNN. In both methods, spatial alignment is not used alone. It is combined with other attention mechanisms. In DANA, a Background Attenuation block (i.e. a global self-attention mechanism) is applied to the support features to highlight class-relevant features and soften background ones. In Meta R-CNN, aligned features are reweighted with global vectors computed from the similarity matrix between query and support features. This last operation may be redundant as the similarity information is already embedded into the aligned features, whereas background attenuation leverages new information.

From this comparison, one can conclude that both global attention and spatial alignment are ben-
eficial for FSOD. However, these improvements may not always be compatible, as shown by the
results of Meta R-CNN. Hence, the design of each component must be done carefully so that spatial
alignment, global attention, and fusion work in unison.

Another set of experiments is conducted on MS COCO dataset. Only the two best-performing meth-
ods on Pascal VOC are selected and trained on MS COCO following the same experimental setup.
The results are summarized in Tab. 6.4. The mAP values are reported following standards from Pas-
cal VOC (mAP<sub>0.5</sub> with one IoU threshold), and MS COCO (mAP<sub>0.5:0.95</sub> with several thresholds). MS
COCO is a much more difficult detection benchmark; therefore the numbers of shots is adjusted to 1,
5, 10, and 30 shots. These results comfort the conclusion obtained on Pascal VOC: the framework is
flexible enough to implement various FSOD techniques that achieve competitive results with state-
of-the-art. As for Pascal VOC the models achieve better performance with more shots. However,
unlike on Pascal VOC, base classes also benefit significantly from a larger number of examples on
<figure><img src="image_75.png" /><figcaption>Figure 6.6: mAP<sub>0.5</sub> on Pascal VOC against the number of shots for each class and each method. Dashed lines represent average performance on all classes, either base classes (top) or novel classes (bottom).</figcaption></figure>

<table><caption>Table 6.4: Performance comparison between WSAAN [229] and DANA [234] on MS COCO. mAP<sub>0.5:0.95</sub> (MS COCO mAP, with IoU thresholds ranging from 0.5 to 0.95) and mAP<sub>0.5</sub> values are reported for base and novel classes separately and for different numbers of shots: K ∈ {1, 5, 10, 30}.</caption><thead><tr><th rowspan="2">K</th><th colspan="4">WSAAN [229]</th><th colspan="4">DANA [234]</th></tr><tr><th colspan="2">mAP<sub>0.5</sub></th><th colspan="2">mAP<sub>0.5:0.95</sub></th><th colspan="2">mAP<sub>0.5</sub></th><th colspan="2">mAP<sub>0.5:0.95</sub></th></tr><tr><th></th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th></tr></thead><tbody><tr><td>1</td><td>33.51</td><td>11.97</td><td>20.12</td><td>6.57</td><td>35.52</td><td>14.54</td><td>21.30</td><td>7.77</td></tr><tr><td>5</td><td>39.88</td><td>19.86</td><td>23.62</td><td>10.54</td><td>42.79</td><td>22.17</td><td>25.19</td><td>11.90</td></tr><tr><td>10</td><td>40.87</td><td>21.42</td><td>24.38</td><td>11.49</td><td>43.00</td><td>23.70</td><td>25.57</td><td>12.89</td></tr><tr><td>30</td><td>41.54</td><td>22.21</td><td>24.74</td><td>12.08</td><td>43.54</td><td>24.36</td><td>25.96</td><td>13.35</td></tr></tbody></table>

MS COCO. MS COCO is more difficult, therefore, the information extracted from the supports better helps the models. Finally, WSAAN outperforms DANA on Pascal VOC but performs slightly worse on MS COCO. It can be noted that the results obtained on a dataset cannot be extrapolated to another without taking into account the characteristics of the datasets. A method that performs best on a dataset is not guaranteed to do so on another dataset. This reinforces the need of a flexible framework that allows fair and easy comparison between FSOD methods. That way, the best-performing method can be easily selected for a given problem. Without such a framework, it is difficult to find out from the literature which method is the most promising for a given application as most FSOD works focus on natural images. For COSE, this framework is highly valuable as it will serve as an objective comparison tool for attention-based FSOD methods.

From these experiments on natural images, it seems clear that DANA performs best. Therefore, it highlights the importance of feature alignment for query-support matching. Global attention loses spatial information in support features which is detrimental to detection. However, global attention methods should not be overlooked. WSAAN shows impressive performance and even outperforms
<table><caption>Table 6.5: Comparison of mAP<sub>0.5</sub> of several methods on DOTA and DIOR datasets. For each method, mAP is reported for different number of shots $K \in \{1, 3, 5, 10\}$ and separately for base and novel classes. Blue and red values represent the best performance on base and novel classes respectively, for each dataset. † denotes results taken directly from the original papers.</caption><thead><tr><th rowspan="3">K</th><th colspan="8">DOTA</th><th colspan="8">DIOR</th></tr><tr><th colspan="2">FRW</th><th colspan="2">WSAAN</th><th colspan="2">DANA</th><th colspan="2">PFRCN</th><th colspan="2">FRW</th><th colspan="2">WSAAN</th><th colspan="2">DANA</th><th colspan="2">PFRCN</th><th colspan="2">WSAAN†</th></tr><tr><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th></tr></thead><tbody><tr><td>1</td><td>47.24</td><td><span style="color:red;">13.35</span></td><td>45.55</td><td>12.19</td><td><span style="color:blue;">49.38</span></td><td>12.52</td><td>34.98</td><td>7.51</td><td>56.67</td><td>16.92</td><td>56.41</td><td>15.48</td><td><span style="color:red;">58.78</span></td><td><span style="color:red;">20.64</span></td><td>40.66</td><td>6.07</td><td>-</td><td>-</td></tr><tr><td>3</td><td>46.50</td><td><span style="color:red;">25.32</span></td><td>44.18</td><td>24.42</td><td><span style="color:blue;">49.67</span></td><td>20.70</td><td>34.58</td><td>9.33</td><td>58.05</td><td>25.08</td><td>51.72</td><td>13.84</td><td><span style="color:red;">59.14</span></td><td><span style="color:red;">27.26</span></td><td>40.48</td><td>7.51</td><td>-</td><td>-</td></tr><tr><td>5</td><td>48.60</td><td>29.57</td><td>47.56</td><td><span style="color:red;">31.44</span></td><td><span style="color:blue;">52.49</span></td><td>24.96</td><td>36.09</td><td>11.33</td><td>60.75</td><td>32.58</td><td>60.79</td><td>30.38</td><td><span style="color:red;">62.12</span></td><td><span style="color:red;">34.16</span></td><td>41.97</td><td>8.55</td><td>-</td><td>0.25</td></tr><tr><td>10</td><td>49.04</td><td>35.29</td><td>46.72</td><td>35.12</td><td><span style="color:blue;">53.99</span></td><td><span style="color:red;">36.50</span></td><td>36.32</td><td>11.55</td><td>61.30</td><td>37.29</td><td><span style="color:blue;">62.79</span></td><td>32.38</td><td>62.71</td><td><span style="color:red;">38.18</span></td><td>42.37</td><td>9.16</td><td>0.54</td><td>0.32</td></tr></tbody></table>

slightly DANA on Pascal VOC. It could be interesting to combine both methods, but this does not seem to be trivial as demonstrated by the results of Meta R-CNN, which leverages the alignment from DANA and the attention from FRW, but does not yield better results.

### 6.2.5.3 Aerial images

To our knowledge, very few works evaluate FSOD methods on aerial images at the time we proposed the AAF framework. Among those we select FSOD-RSI [225], which simply applies FRW to aerial images (we will refer to it as FRW), WSAAN [223] and our PFRCNN. In addition, we include DANA inside this comparison as it was the best-performing technique on natural images. All these methods are evaluated on different datasets, making their performance comparison challenging. Using the proposed AAF framework, we compare the performance of these methods on both DOTA and DIOR. These methods are reimplemented inside the framework and all other design choices are fixed during the experiments (as described in Sec. 6.2.4). Tab. 6.5 regroups the results of the comparison. These results show a slight improvement over the state-of-the-art on DIOR (WSAAN [223]). Our implementation of WSAAN outperforms (8 mAP<sub>0.5</sub> points on base classes and 0.4 on novel classes) the result reported in the original paper. However, the attention mechanism employed in WSAAN is not optimal for aerial images. WSAAN is outperformed by both FRW and DANA. While this was expected for DANA in the light of results from Sec. 6.2.5.2, it was not for FRW. The superiority of DANA over the other methods on DOTA and DIOR is clear and coherent with the results on natural images. The more sophisticated attention mechanism, in particular the alignment, from DANA is better at extracting and leveraging the information from the support examples. Hence, the detection performance is higher. It is particularly beneficial for small numbers of shots: the extracted information is semantically robust.

These results confirm the analysis conducted in Chap. 4, the performance gap between the classical baseline (i.e. FCOS) and the few-shot approaches is larger on aerial images. On natural images, the performance drop between the few-shot approach and the regular baseline is nearly inexistent for base classes and around 25% for novel classes. On aerial images these drops are largely increased: ~ 15% and ~ 50% for base and novel classes respectively. This can be guessed from Tabs. 6.3 and 6.5, but detailed gaps are provided in Tab. 6.6. Following the analysis from Chap. 4, the main reason behind this performance gap between natural and aerial images is the size of the objects in
the image. Small objects are much more difficult to detect and also are poor representatives of a semantic class as they contain little information. Therefore, the most sensible direction to pursue is to design new attention mechanisms specifically built for small objects. As it happens, this will be discussed in the next section.

<table><caption>Table 6.6: mAP<sub>0.5</sub> and RmAP values for some reimplemented methods and XQSA with K = 10 shots.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="2">DOTA</th><th colspan="2">DIOR</th><th colspan="2">Pascal VOC</th></tr><tr><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th></tr></thead><tbody><tr><td rowspan="4">mAP<sub>0.5</sub></td><td>FCOS baseline</td><td>60.87</td><td>69.69</td><td>72.82</td><td>81.48</td><td>65.47</td><td>68.02</td></tr><tr><td>FRW</td><td>49.04</td><td>35.29</td><td>61.30</td><td>37.29</td><td>63.16</td><td>48.71</td></tr><tr><td>WSAAN</td><td>46.72</td><td>35.12</td><td><span style="color:blue;">62.79</span></td><td>32.38</td><td><span style="color:blue;">65.27</span></td><td>51.70</td></tr><tr><td>DANA</td><td><span style="color:blue;">53.99</span></td><td><span style="color:red;">36.50</span></td><td>62.71</td><td><span style="color:red;">38.18</span></td><td>65.17</td><td><span style="color:red;">52.26</span></td></tr><tr><td rowspan="4">RmAP (%)</td><td>FRW</td><td>-19.43</td><td>-49.36</td><td>-15.83</td><td>-54.23</td><td>-3.52</td><td>-28.39</td></tr><tr><td>WSAAN</td><td>-23.24</td><td>-49.60</td><td>-13.78</td><td>-60.26</td><td>-0.30</td><td>-24.00</td></tr><tr><td>DANA</td><td>-11.30</td><td>-47.63</td><td>-13.88</td><td>-53.14</td><td>-0.46</td><td>-23.17</td></tr></tbody></table>

## 6.3 Cross-Scales Query-Support Alignment for Small FSOD

From the analysis in Chap. 4 and the previous section, it is clear that a new attention mechanism specifically designed for small objects is required to get reasonable performance on aerial images. To this end, we propose a novel alignment method that combines information from multiple scales: Cross-Scales Query-Support Alignment (XQSA). This differs from existing methods which often work independently at different scales. Conversely, XQSA combines the information from various scales and sources (i.e., query and support images). Its original motivation is to unlock matching support examples with query objects belonging to the same class even though their sizes differ. With existing methods, this was prohibited as same-class objects with different sizes have non-similar features and are not matched by similarity-based attention mechanisms.

### 6.3.1 XQSA definition

In this section, we detail the functioning of our proposed Cross-Scales Query-Support Alignment module. First, features are extracted from the query and support images with a backbone network $f$. In our implementation, $f$ is a ResNet-50 with an FPN attached. It outputs feature maps at three distinct resolution levels:

$$ \{F_{q,0}, F_{q,1}, F_{q,2}\} = f(I_q), \qquad (6.6) $$

$$ \{F_{s,0}^c, F_{s,1}^c, F_{s,2}^c\} = f(I_s^c). \qquad (6.7) $$

All query features $F_{q,i} \in \mathbb{R}^{w_{q,i} \times h_{q,i} \times d}$, for $i \in \{0, 1, 2\}$ (i.e. from different levels) are flattened and concatenated into a unique representation $F_q \in \mathbb{R}^{n_q \times d}$, with $n_q = \sum_i w_{q,i}h_{q,i}$. Here, $w_{q,i}$ and $h_{q,i}$
denote the size of the query feature map at level *i*, this size depends on the query image size and the stride of the corresponding level in the backbone. The same operation is performed for all support features $F_{s,i}^c$. When more than one shot is available per class, support features are averaged at each level *i*:

$$F_{s,i}^c = \frac{1}{K} \sum_{k=1}^{K} F_{s,i}^{c,k}. \qquad (6.8)$$

Then, following the ViT paradigm, the support and query features are linearly projected into the queries, keys and values matrices $Q$, $K$ and $V$. Specifically, the query features are used to produce the queries while keys and values are computed from the support features:

<figure><img src="image_76.png" /></figure>

where $W_Q$, $W_K$ and $W_V$ are learnable projection matrices, which are implemented as linear layers in practice. From this, an affinity matrix is computed between the queries and the keys, and then the aligned support features $A_s^c$ are computed as:

$$\lambda_s^c = \text{Softmax}(\frac{QK^{cT}}{\sqrt{d}}), \qquad (6.12)$$

$$A_s^c = \lambda_s^c V^c. \qquad (6.13)$$

For completeness with the definition of the AAF framework, $\lambda_q = I$, meaning that the query features are not modified. The aligned features are finally processed by a two-layer MLP with skip connections. LayerNorm [301] is applied before alignment and the MLP. These supplementary computations can be seen as fusion operations in the AAF framework, similar to what was proposed in [227, 222, 249]. This resembles the ViT attention, but with a major difference, it combines features from different images and different levels (see Fig. 6.7). This allows better object matching when there are size discrepancies between support and query images.

Small objects have a limited footprint in feature maps which make them hard to detect but also hard to match with support examples. XQSA's multiscale alignment enhances the chances of matching as each query feature is compared with support features at all scales. Finally, in order to fairly compare XQSA with DANA, we leverage their BackGround Attenuation block (BGA) on the support features before alignment. They conduct a thorough ablation study which shows the positive impact of BGA on the few-shot performance of their method. We also carry out an ablation study about our cross-
<figure><img src="image_77.png" /><figcaption>Figure 6.7: Diagram illustrating the proposed cross-scales query-support alignment method. Features are extracted from the query and support images at multiple scales and combined to form an affinity matrix. For each query feature position, the affinity is computed with any position in the support features. This allows object matching across different feature levels.</figcaption></figure>

scales method (see Tab. 6.7) and find that BGA also improves performance in this case. XQSA is implemented inside the AAF framework, split into three modules: alignment, attention and fusion, following the description from Sec. 6.2. BGA is implemented within the global attention block, which can handle as well self-attention module as well and is applied before alignment.

### 6.3.2 Ablation study XQSA

To confirm the benefits of each component of our attention methods, we conduct a brief ablation experiment, adding separately the different modules of our proposed attention mechanism. The ablation is conducted on DOTA and the results are available in Tab. 6.7. From this table, it is clear that each component plays a role in the improved performance of our method. Both the fusion (with the MLP) and the skip connections around fusion and alignment are beneficial for the performance on novel classes. It is worth noting that Background Attenuation proposed by [234] helps both for base and novel classes, which confirms the experiments conducted by the authors of this work.

<table><caption>Table 6.7: Ablation study of the XQSA attention method on DOTA dataset. mAP<sub>0.5</sub> scores are reported for base and novel classes with *K* = 10 shots.</caption><tr><td>Baseline</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Cross-scale Alignment</td><td></td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Fusion Layer</td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><td>Query-Support Self-Attention</td><td></td><td></td><td></td><td>✓</td></tr><tr><td>Base classes</td><td>49.20</td><td>49.46</td><td>49.13</td><td>51.11</td></tr><tr><td>Novel classes</td><td>36.52</td><td>38.84</td><td>40.31</td><td>41.01</td></tr></table>
<table><caption>Table 6.8: Performance comparison between XQSA, FRW, and DANA. mAP<sub>0.5</sub> values are reported separately for base (top) and novel (bottom) classes on DOTA, DIOR, Pascal VOC, and MS COCO with K = 10 shots. mAP values are reported for All, Small (√wh < 32), Medium (32 ≤ √wh < 96) and Large (√wh ≥ 96) objects.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="4">DOTA</th><th colspan="4">DIOR</th><th colspan="4">Pascal VOC</th><th colspan="4">MS COCO</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td rowspan="3">Base Classes</td><td>FRW</td><td>49.04</td><td>25.48</td><td>59.17</td><td>63.37</td><td>62.20</td><td>8.21</td><td>48.66</td><td>80.67</td><td>63.21</td><td>15.67</td><td>47.94</td><td><b>81.73</b></td><td>29.03</td><td>13.08</td><td>35.87</td><td>48.00</td></tr><tr><td>DANA</td><td><b>53.99</b></td><td><b>36.98</b></td><td><b>62.33</b></td><td><b>70.39</b></td><td><b>62.71</b></td><td><b>10.92</b></td><td><b>49.34</b></td><td><b>83.17</b></td><td><b>65.17</b></td><td><b>18.14</b></td><td><b>50.58</b></td><td><b>80.11</b></td><td><b>38.14</b></td><td><b>23.30</b></td><td><b>51.85</b></td><td><b>56.38</b></td></tr><tr><td>XQSA</td><td>51.11</td><td>26.10</td><td>59.41</td><td>64.30</td><td>59.88</td><td>10.64</td><td>45.69</td><td>82.34</td><td>62.13</td><td>15.60</td><td>48.64</td><td>75.94</td><td>31.56</td><td>16.13</td><td>40.13</td><td>49.83</td></tr><tr><td rowspan="3">Novel Classes</td><td>FRW</td><td>35.29</td><td>13.99</td><td>34.11</td><td>59.31</td><td>37.29</td><td>2.48</td><td>33.74</td><td>59.38</td><td>48.72</td><td>16.44</td><td>26.71</td><td><b>68.27</b></td><td>24.09</td><td>11.53</td><td>22.45</td><td><b>38.69</b></td></tr><tr><td>DANA</td><td>36.58</td><td>14.32</td><td>40.28</td><td><b>64.65</b></td><td>38.18</td><td>3.21</td><td>34.91</td><td><b>60.99</b></td><td>52.26</td><td>10.05</td><td>24.67</td><td>67.23</td><td>24.75</td><td>12.01</td><td><b>29.40</b></td><td>37.95</td></tr><tr><td>XQSA</td><td><b>41.00</b></td><td><b>17.84</b></td><td><b>44.57</b></td><td>54.46</td><td><b>41.51</b></td><td><b>4.12</b></td><td><b>40.69</b></td><td>58.21</td><td><b>53.94</b></td><td><b>19.46</b></td><td><b>34.86</b></td><td>66.14</td><td><b>25.03</b></td><td><b>12.57</b></td><td>26.05</td><td>38.55</td></tr></tbody></table>

### 6.3.3 Application to aerial and natural images

To assess the capabilities of the proposed method, we compare it with the best methods from Sec. 6.2.5: FRW and DANA on DOTA, DIOR, Pascal VOC and MS COCO. The results of these experiments are available in Tab. 6.8. The mAP values are reported separately for small (√wh < 32), medium (32 ≤ √wh < 96) and large (√wh ≥ 96) objects. Hence, the methods can be compared specifically on small objects. XQSA performs consistently better on small and medium novel objects, compared with FRW and DANA. This performance gain comes at the expense of slightly lower detection quality for larger objects. In XQSA, the shallow query features are compared to all support features (i.e., not only shallow support maps). As deeper maps are smaller, this increases moderately the number of potential detections for small objects. However, for large objects, the deep query feature map is compared with all support maps, including the shallow ones. It increases a lot the number of potential matches between query and support features (see an illustration of this phenomenon in Fig. 6.8). For large objects that are already well detected, this mostly adds wrong matches and deteriorates the performance. For small objects, however, this is useful as very few correct matches are found by current FSOD methods. A potential solution for this issue would be to down-weight the contributions of shallower features in the affinity matrix’s bottom rows (i.e., the left and bottom blocks of the matrix). The affinity matrix could even be made upper triangular to avoid taking into account the contributions of shallower levels at all. For similar reasons, XQSA demonstrates a slight drop in base classes performance. However, the actual goal of few-shot learning is to maximize performance on novel classes. The large number of available examples during base training is enough to learn robust query-support matching even for small objects. However, our goal here is to improve the generalization capabilities of the model on novel objects. The performance on base classes is simply a safety check and relates more to the Generalized FSOD problem (see Sec. 3.4).

Looking at the performance on all objects, disregarding their size, the proposed alignment technique significantly improves the detection quality for aerial images. Using XQSA in the AAF framework increased novel class mAP by 5 and 4 points on DOTA and DIOR, respectively. As it works better on small objects but worse on large objects, it is less appropriate for natural images. As a consequence, it shows lower improvements for Pascal VOC and MS COCO. Overall, XQSA largely improves on
<figure><img src="image_78.png" /><figcaption>Figure 6.8: XQSA small and large objects matching asymmetry.</figcaption></figure>

<table><caption>Table 6.9: Performance comparison between XQSA, FRW, and DANA. mAP<sub>0.5:0.95</sub> values are reported separately for base (top) and novel (bottom) classes on DOTA, DIOR, Pascal VOC, and MS COCO with *K* = 10 shots. mAP values are reported for All, Small (√wh < 32), Medium (32 ≤ √wh < 96) and Large (√wh ≥ 96) objects.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="4">DOTA</th><th colspan="4">DIOR</th><th colspan="4">Pascal VOC</th><th colspan="4">MS COCO</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td rowspan="3">Base Classes</td><td>FRW</td><td>23.18</td><td>8.60</td><td>27.84</td><td>32.22</td><td>35.60</td><td>2.60</td><td>23.04</td><td>50.82</td><td>37.93</td><td>6.54</td><td>22.84</td><td>50.49</td><td>15.60</td><td>5.47</td><td>18.84</td><td>27.83</td></tr><tr><td>DANA</td><td>26.63</td><td>11.43</td><td>30.73</td><td>37.62</td><td>36.39</td><td>3.48</td><td>24.93</td><td>52.31</td><td>39.12</td><td>7.28</td><td>25.37</td><td>51.39</td><td>22.46</td><td>10.22</td><td>29.72</td><td>36.51</td></tr><tr><td>XQSA</td><td>25.30</td><td>8.85</td><td>28.78</td><td>34.64</td><td>34.80</td><td>3.54</td><td>22.90</td><td>51.47</td><td>27.45</td><td>3.18</td><td>16.60</td><td>36.76</td><td>11.37</td><td>4.44</td><td>14.18</td><td>31.97</td></tr><tr><td rowspan="3">Novel Classes</td><td>FRW</td><td>15.99</td><td>4.25</td><td>14.09</td><td>29.65</td><td>20.00</td><td>0.48</td><td>17.00</td><td>33.30</td><td>29.09</td><td>5.64</td><td>12.21</td><td>40.03</td><td>12.41</td><td>4.84</td><td>10.90</td><td>20.82</td></tr><tr><td>DANA</td><td>17.17</td><td>5.60</td><td>20.44</td><td>32.40</td><td>20.35</td><td>0.78</td><td>17.49</td><td>34.01</td><td>31.75</td><td>5.23</td><td>11.09</td><td>43.38</td><td>13.44</td><td>5.30</td><td>15.03</td><td>21.47</td></tr><tr><td>XQSA</td><td>21.04</td><td>7.91</td><td>25.18</td><td>26.49</td><td>22.78</td><td>0.97</td><td>20.97</td><td>34.78</td><td>25.07</td><td>6.40</td><td>12.74</td><td>35.15</td><td>10.33</td><td>4.87</td><td>10.04</td><td>16.72</td></tr></tbody></table>
<figure><img src="image_79.png" /><figcaption>Figure 6.9: mAP<sub>0.5</sub> and corresponding RmAP values of the four best performing methods from all our experiments. All methods are trained within our proposed AAF framework with data augmentation which explains slightly higher performance for FRW and WSA. 10 shots are available for each class at inference time.</figcaption></figure>

existing works for aerial images. On DIOR, this corresponds to a 10 mAP point increase compared to previous state-of-the-art [223]. However, this is not sufficient to fill the performance gap with natural images as presented in Fig. 6.9. This figure extends Fig. 4.2 with XQSA results. While XQSA improves upon other methods on aerial images, it is still far behind the performance of the regular baseline. XQSA is better for small and medium objects but at the cost of lower performance on large objects and base classes. Progress is still required to get more versatile FSOD solutions able to handle small, medium, and large objects at the same time.

In complement to Tab. 6.8, we also provide the comparison between XQSA, DANA and FRW with mAP<sub>0.5:0.95</sub> metric and on the four datasets. These results are provided in Tab. 6.9. mAP<sub>0.5:0.95</sub> is a more demanding metric for object detection. It is especially hard for small objects as a few pixels shift from ground truth can greatly reduce the IoU and therefore lead to a missed detection. This intensifies as the IoU threshold increases in the mAP computation.

For DOTA and DIOR, results with mAP<sub>0.5:0.95</sub> are in agreement with results from Tab. 6.8 (i.e. with mAP<sub>0.5</sub>). However, XQSA does not perform better than DANA on Pascal VOC and MS COCO novel classes with mAP<sub>0.5:0.95</sub>. This is mainly due to the metric being too strict on small objects. This questions the soundness of these metrics for FSOD, especially when dealing with small objects. We will tackle this question in Chap. 8.
### 6.3.4 XQSA Implementation Challenges and Extensions

XQSA is inspired by the ViT and resembles some FSOD techniques that leverage transformer attention as well. It is well-known that these mechanisms are computationally heavy and scale quadratically in terms of the number of locations in the feature maps. Here, it is slightly different as we combine the features from the support and query images. Support images are 4× smaller than query images (see Sec. 6.2.4.1), therefore the complexity of the attention module is greatly reduced. According to the notations introduced in Sec. 6.3.1, the complexity of the XQSA alignment block is $O(n_q n_s)$. As $n_s$ is way smaller than $n_q$ (roughly 16 times), this is way better than a self-attention mechanism which would be $O(n_q^2)$. However, with XQSA, as features from all scales are concatenated, this remains computationally heavy, especially as this process must be repeated for each support image. Unfortunately, this does not scale well as we increase the number of support examples. Time complexity rapidly becomes prohibitive, but memory complexity is more limiting. The gradients of the Softmax used for the computation of the similarity matrices are extremely large ($O(n_q n_s^2)$) and do not fit on GPU memory when the number of support examples increases. To bypass these limitations, we propose several tricks.

- **Pytorch manual gradient computation**: automatic differentiation in Pytorch is not always optimal. When successive computations involve the same gradients, Pytorch often computes and stores them separately, wasting precious resources. To this end, we re-implemented the XQSA block as a self-contained operation, with custom gradients computation to prevent duplicated gradients. This results in slight memory and performance gains, but it is still not enough to scale efficiently with the number of support examples.

- **Pytorch gradient checkpointing**: Pytorch has an API to checkpoint the gradients during the backward pass. It copies back the gradients on the CPU memory to prevent overflowing the GPU memory. It solves the out-of-memory issues, but makes the training much slower.

- **Deformable XQSA block**: The alignment block is expensive due to the comparison between all query feature locations and all support locations. Thus, we extend the XQSA block with a deformable attention mechanism, inspired from ConViT [65]. Specifically, we introduce an inductive bias inside the attention module by adding a layer that selects the locations that will be compared between support and query feature maps. This resembles Deformable Convolutions [61] and Deformable-DETR [60]. While this was solving both the memory overflow and training slowness, we were not able to achieve reasonable detection performance with it.

- **Support class averaging**: while this solution seems sub-optimal, it saves a lot of time and memory by avoiding a lot of computation. Of course, it does not completely solve the scaling issues (e.g., as the number of support classes increases). However, it allows adding a large number of support images per class without any issues and performs well on the FSOD task.

Finally, we only keep the support class averaging as it is the simplest and best-performing alternative tested.
<figure><img src="image_80.png" /><figcaption>Figure 6.10: Support examples for base classes (K = 10). These are the examples used during the detection inference that produced results from Fig. 6.13.</figcaption></figure>

**6.4 Qualitative Comparison within the AAF framework**

In this section, we provide a qualitative comparison of the four attention-based methods that we compared previously in Fig. 6.9: FRW, WSAAN, DANA and ours XQSA. To get a fair comparison, we sampled the same support examples for each method and performed the detection on the same query images. For convenience, we split the comparison in two, first on base classes and then on novel classes. In both cases, we provide the support examples used for the detection in the first figure and the detection on a handful of images in the second figure. These comparisons are visible in Figs. 6.10 and 6.11 for base classes and in Figs. 6.12 and 6.13 for novel classes.

**6.4.1 Base Classes Detection Quality**

From Fig. 6.11, it is quite difficult to assess which method is superior to the other. It seems quite obvious that DANA and XQSA perform slightly better than FRW and WSAAN. However, there are cases where neither DANA nor XQSA is the best (see the second row for instance). This is in line with the quantitative results from the previous sections. The gap between these methods on base classes is tight, it is therefore quite difficult to correctly assess the quality of these methods from only a handful of examples. It is worth noting that these techniques work quite well on the base classes. Of course, there are some false positives, but the number of false negatives is limited, which is quite important for intelligence applications.

**6.4.2 Novel Classes Detection Quality**

With novel classes, the performance differences are more visible than with base classes. First, it can be seen that XQSA produces less false positives which indicates a higher accuracy. Then, XQSA also provides more small detection than other methods. It sometimes gives false positives, but overall it improves the detection of small targets (see the last row for instance). Interestingly, XQSA and DANA are less sensitive to partial objects in the images (see first row). This may be explained by the spatial information kept in the query-support combination compared to FRW and WSAAN. As the whole object is available in the support examples, DANA and XQSA match all parts of the support
<figure><img src="image_81.png" /><figcaption>Figure 6.11: Qualitative assessment of the detection quality of FRW, WSAAN, DANA and XQSA on DOTA with *K* = 10 shots on base classes.</figcaption></figure>
features with the query features and detect only entire objects. It seems that this kind of matching
is much more robust than the global attention alone (FRW and WSAAN) as they showcase many
more false positives overall. Of course, the detection of small objects is still very challenging in the
FSOD setting which explains the relatively poor detection quality in these images. It is not easy
to objectively determine which method is the best at this, but a slight advantage seems in favor of
XQSA, confirming quantitative results.

**6.5 Conclusion**

In this chapter, we have introduced a highly modular framework for implementing attention-based FSOD methods. First, this framework allows fair comparison between the various attention mechanisms proposed in the literature. From our analysis, it seems that spatial alignment is crucial to achieving high-quality FSOD, mostly because it does not lose the spatial information contained in the support examples. Secondly, the AAF framework is a practical tool to design new attention mechanisms. For that matter, we developed a novel cross-scales alignment layer within the framework to specifically increase the detection performance on the small objects. The so-called XQSA alignment allows us to achieve large improvements compared to the contemporary literature on several datasets. It works especially well on aerial images as they contain smaller and more objects than natural images. Specifically, XQSA outperforms the state-of-the-art on DOTA and DIOR datasets at that time.

Nevertheless, the attention-based methods are not completely satisfactory from an industrial perspective. While they achieve reasonable performance on aerial images, they have some disadvantages. First, even if they can adapt to novel classes from a few support examples at test time, they still require extensive fine-tuning to perform correctly. This fine-tuning can take up to several hours, which is not convenient for "on-the-fly" adaptation. Then, the episodic training strategy is somewhat cumbersome and generates unrealistic scenarios. Indeed, during each episode, query images are sampled so that they contain at least one instance of one of the episode classes. In real-case applications, no object of interest can be visible in an image which makes the detection task more challenging. But most importantly, only the episode classes are detected during the episode, the detection task is, therefore, simpler as it only classifies objects among a smaller number of classes. Given these drawbacks, we investigate in the next chapter FSOD methods that do not employ the episodic training strategy and therefore solely rely on a fine-tuning scheme.
<figure><img src="image_82.png" /><figcaption>Figure 6.12: Support examples for novel classes ($K = 10$). These are the examples used during the detection inference that produced results from Fig. 6.13.</figcaption></figure>
<figure><img src="image_83.png" /><figcaption>Figure 6.13: Qualitative assessment of the detection quality of FRW, WSAAN, DANA and XQSA on DOTA with *K* = 10 shots on novel classes.</figcaption></figure>
CHAPTER 7

FEW-SHOT DIFFUSION DETECTOR VIA FINE-
TUNING

ABSTRACT

Previous chapters explore few-shot object detection with metric learning and attention-based techniques. This chapter focuses on the last major approach for FSOD: fine-tuning. Based on DiffusionDet, a recent detection framework leveraging diffusion models, we build a simple but efficient fine-tuning strategy. The resulting method, called FSDiffusionDet, achieves state-of-the-art FSOD on aerial datasets and competitive performance on natural images. Extensive experimental studies explore the design choices of the fine-tuning strategy to better understand the key components required to achieve such quality. Finally, these impressive results allow considering more complex settings such as cross-domain scenarios, which are especially relevant for COSE.

CONTENTS

<table>
  <tbody>
    <tr>
      <td>7.1</td>
      <td>Diffusion Probabilistic Models Principle.</td>
      <td>134</td>
    </tr>
    <tr>
      <td>7.1.1</td>
      <td>Forward and Reverse Diffusion Processes</td>
      <td>134</td>
    </tr>
    <tr>
      <td>7.1.2</td>
      <td>DDPM Training and Sampling</td>
      <td>136</td>
    </tr>
    <tr>
      <td>7.1.3</td>
      <td>Recent Advances with Diffusion Models.</td>
      <td>139</td>
    </tr>
    <tr>
      <td>7.2</td>
      <td>DiffusionDet for Object Detection.</td>
      <td>141</td>
    </tr>
    <tr>
      <td>7.3</td>
      <td>Few-Shot DiffusionDet.</td>
      <td>143</td>
    </tr>
    <tr>
      <td>7.3.1</td>
      <td>Fine-tuning Strategy for Few-Shot DiffusionDet.</td>
      <td>143</td>
    </tr>
    <tr>
      <td>7.3.2</td>
      <td>Experimental Study of FSDiffusionDet.</td>
      <td>145</td>
    </tr>
    <tr>
      <td>7.3.3</td>
      <td>Comparison with existing FSOD Methods</td>
      <td>157</td>
    </tr>
    <tr>
      <td>7.4</td>
      <td>Application to Cross-Domain FSOD.</td>
      <td>160</td>
    </tr>
    <tr>
      <td>7.4.1</td>
      <td>MS COCO → Anything.</td>
      <td>160</td>
    </tr>
    <tr>
      <td>7.4.2</td>
      <td>Aerial Cross-Domain</td>
      <td>164</td>
    </tr>
    <tr>
      <td>7.4.3</td>
      <td>Cross-Domain Perspectives.</td>
      <td>165</td>
    </tr>
    <tr>
      <td>7.5</td>
      <td>Conclusion</td>
      <td>166</td>
    </tr>
  </tbody>
</table>
In Chaps. 5 and 6 we have proposed respectively metric-learning and attention-based approaches to tackle the FSOD problem. Both of these directions were sensible choices given the state of the FSOD literature at the beginning of this project. Since then, however, fine-tuning approaches have gained a lot of interest with competitive performance and reduced complexity. Following this trend, we explore in this chapter a simple fine-tuning strategy for FSOD. Based on the recent DiffusionDet [74] model, we propose an effective fine-tuning scheme for FSOD which outperforms all previous methods on DOTA and DIOR datasets while being competitive with state-of-the-art on natural images. We begin with a brief presentation of the Diffusion Probabilistic Models (DPM) and their recent progress in various generative tasks. Then we present in detail DiffusionDet, which tackles OD with a refreshing perspective, as a box-denoising problem. Following this, we present our fine-tuning strategy called Few-Shot DiffusionDet (FSDiffusionDet) and the results of multiple experiments conducted to improve our strategy. Given the impressive performance of FSDiffusionDet in the few-shot regime, we broaden the scope of our analysis and study the more challenging Cross-Domain FSOD task. We emphasize that the first two sections of this chapter present existing works in the literature, while the last three sections discuss our contributions: the FSDiffusionDet strategy, thorough experimental analysis of the strategy on several datasets, and its application in Cross-Domain scenarios.

## 7.1 Diffusion Probabilistic Models Principle

Diffusion Probabilistic Models have been introduced in 2015 by Sohl-Dickstein et al. [199]. Their principle is simple, to approximate a complex and intractable probability distribution, they model a diffusion process from the original distribution to a normal distribution as gradual Gaussian noise addition. Then, the goal is to find the reverse process to approximate the original distribution by iterative denoising. In this section, we present the main concepts of the DPM [199] and their recent advances in generative tasks, mostly led by Denoising Diffusion Probabilistic Models (DDPM) [200].

### 7.1.1 Forward and Reverse Diffusion Processes

First, let's introduce a few notations. Our objective is to be able to efficiently sample elements $x$ from a distribution $P$. When $P$ is an arbitrary distribution, its Probability Density Function (PDF) is intractable and sampling often relies on expensive Monte Carlo techniques. Here, we suppose that there exists a random process $q$ that transforms $P$ into a normal distribution:

$$q(x) \sim \mathcal{N}(\mathbf{0}, \mathbf{I}). \qquad (7.1)$$

This is called the Diffusion Process and refers to the eponym physical phenomenon. The main hypothesis is to assume that this process is a Markov Chain that adds Gaussian noise progressively:

$$x_T = q(x_{1:T}|x_0), \qquad (7.2)$$
where $x_0 \sim \mathcal{P}$ is an element sampled from the original distribution and $x_T \sim \mathcal{N}(0, I)$ is sampled from a normal distribution. $T$ denotes here the number of steps in the Markov chain and $q(x_{1:T})$ represents the joint distribution of variables $x_1$ to $x_T$. The diffusion process adds Gaussian noise iteratively and therefore is defined as:

$$ q(x_{1:T}|x_0) := \prod_{t=1}^{T} q(x_t|x_{t-1}) \quad \text{with} \quad q(x_t|x_{t-1}) := \mathcal{N}(\sqrt{1-\beta_t}x_t, \beta_t I). \qquad (7.3) $$

The $\beta_t$ denote the variance schedule, i.e., the amount of gaussian noise added at each step. For convenience, we also define $\alpha_t$ and $\bar{\alpha}_t$:

$$ \alpha_t = 1 - \beta_t, \qquad (7.4) $$

$$ \bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s. \qquad (7.5) $$

$q$ is called the *forward* diffusion process as it transforms $x_0$ into noise (this is true only asymptotically when $T \to \infty$). Thus, we can write for $1 \le t \le T$:

$$ x_t = \sqrt{\alpha_t} x_{t-1} + \sqrt{1-\alpha_t} \epsilon_{t-1}, \qquad (7.6) $$

$$ = \sqrt{\alpha_t \alpha_{t-1}} x_{t-2} + \sqrt{1-\alpha_t \alpha_{t-1}} \epsilon_{t-2}, \qquad (7.7) $$

$$ = \dots = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon_0, \qquad (7.8) $$

where the $\epsilon_i$ ($0 \le i \le t-1$) are sampled from a normal distribution.

The *reverse process* instead transforms Gaussian noise into elements sampled from $\mathcal{P}$. It is also a Markov chain with Gaussian transitions (this is ensured for sufficiently small $\beta_t$):

$$ q(x_{0:T}) := \prod_{t=1}^{T} q(x_{t-1}|x_t) \quad \text{with} \quad q(x_{t-1}|x_t) := \mathcal{N}(\mu_t, \Sigma_t). \qquad (7.9) $$

Unfortunately, the reverse process, or rather $\mu_t$ and $\Sigma_t$, are highly intractable and cannot be easily estimated. However, we can approximate this process with a parametrized model $p_\theta$:

$$ p_{\theta}(x_{0:T}) := p(x_T) \prod_{t=1}^{T} p_{\theta}(x_t|x_{t-1}) \quad \text{with} \quad p_{\theta}(x_t|x_{t-1}) := \mathcal{N}(\mu_{\theta}(x_t, t), \Sigma_{\theta}(x_t, t)). \qquad (7.10) $$

Hence, if we find an optimal set of parameters $\theta$ so that the model is able to approximate the reverse process expectation $\mu_t$ and variance $\Sigma_t$ from variable $x_t$ and timestep $t$, then the reverse process can be computed. Now, we must derive efficient ways to estimate the reverse process.
One solution is to leverage deep neural networks which are well suited for these kinds of tasks. We will explain how such models can be trained to approximate the reverse process conditional probabilities $q(x_{t-1}|x_t)$. Note that the reverse process is initialized with a normal distribution: $p(x_T) = N(0, I)$. The forward and reverse diffusion processes are illustrated in Fig. 7.1, in the context of image denoising. Before jumping to the next section to see how we can estimate such models, we can observe that the reverse process distribution is tractable when conditioned on $x_0$, it will be useful for training:

$$q(x_{t-1}|x_t, x_0) := N(\tilde{\mu}_t(x_t, x_0), \tilde{\beta}_t I), \qquad (7.11)$$

$$\text{with } \tilde{\mu}_t(x_t, x_0) = \frac{\sqrt{\bar{\alpha}_{t-1}\beta_t}}{1 - \bar{\alpha}_t} x_0 + \frac{\sqrt{\alpha_t(1 - \bar{\alpha}_{t-1})}}{1 - \bar{\alpha}_t} x_t, \qquad (7.12)$$

$$\text{and } \tilde{\beta}_t = \frac{1 - \hat{\alpha}_{t-1}}{1 - \hat{\alpha}_t} \beta_t. \qquad (7.13)$$

<figure><img src="image_84.png" /><figcaption>Figure 7.1: Diffusion Processes illustration in the context of image denoising. Image taken from [200, 302].</figcaption></figure>

### 7.1.2 DDPM Training and Sampling

Now that we have introduced the forward and reverse diffusion processes, we can dive into the core of DDPM and see how we can train models to approximate the reverse process. The overall goal is to maximize the log-likelihood of the modeled data distribution $p_\theta(x_0)$:

$$\theta^* = \arg\max_{\theta} \log(p_{\theta}(x_0)). \qquad (7.14)$$

Unfortunately, the log-likelihood is not easily optimizable and several tricks are required to get a more tractable objective. First, [199] makes use of the well-known Evidence Lower BOund [303] (ELBO), which is lower bound to log-likelihood objective and more easily computable. In practice,
we minimize the negative log-likelihood and leverage the evidence upper bound:

$$-\log(p_{\theta}(x_{1:T})) = -\log \left( \int p_{\theta}(x_{0:T})dx_{1:T} \right), \qquad (7.15)$$

$$= -\log \left( \int p_{\theta}(x_{0:T}) \frac{q(x_{1:T}|x_0)}{q(x_{1:T}|x_0)} dx_{1:T} \right), \qquad (7.16)$$

$$= -\log \mathbb{E}_q \left[ \frac{p_{\theta}(x_{0:T})}{q(x_{1:T}|x_0)} \right], \qquad (7.17)$$

$$\leq \mathbb{E}_q \left[ -\log \frac{p_{\theta}(x_{0:T})}{q(x_{1:T}|x_0)} \right] = \mathcal{L}_{\text{ELBO}}. \qquad (7.18)$$

Now, using the definition of the process $p_{\theta}(x_{0:T})$ and $q(x_{1:T}|x_0)$, and Bayes' rule, we can split the objective for each timestep $t$ and make it tractable:

$$\mathcal{L}_{\text{ELBO}} = \mathbb{E}_q \left[ -\log \frac{p(x_T)}{q(x_T|x_0)} + \sum_{t>1} \left( -\log \frac{p_{\theta}(x_{t-1}|x_t)}{q(x_t|x_{t-1},x_0)} - \log p_{\theta}(x_0|x_1) \right) \right]. \qquad (7.19)$$

Objective for timestep $T$, $\mathcal{L}_T$

Objective for timestep $t-1$, $\mathcal{L}_{t-1}$

$\mathcal{L}_0$

The previous equation is only valid because $q(x_{0:T})$ if a Markov chain and because the Markov Property states that $q(x_t|x_{t-1}, x_0) = q(x_t|x_{t-1})$. This is crucial, otherwise, the Bayes' rule introduces intractable terms ($q(x_t)$ and $q(x_{t-1})$). We refer the reader to Appendix A from [200] and Appendix B from [199] for the detailed derivation of Eq. (7.19). Finally, the terms highlighted in gray and blue in the above Eq. (7.19) can be interpreted as Kullback-Leiber Divergence terms:

$$\mathcal{L}_{\text{ELBO}} = \mathbb{D}_{\text{KL}}(q(x_T|x_0)||p(x_T)) + \sum_{t>1} \mathbb{D}_{\text{KL}}(q(x_t|x_{t-1}, x_0)||p_{\theta}(x_{t-1}|x_t)) \qquad (7.20)$$

$\mathcal{L}_T$

$$+ \mathbb{E}_q \left[ -\log p_{\theta}(x_0|x_1) \right]. \qquad \mathcal{L}_{t-1}$$

$\mathcal{L}_0$

These KL divergence terms are easy to compute as they compare only Gaussian distributions. This gives an easy-to-optimize upper bound to train the diffusion models. Diffusion models are meant to approximate the reverse diffusion process. One way to achieve this with neural network models is to use the reparametrization trick introduced in [303]. The idea is to train a neural network to output the mean and variance parameters of a Gaussian distribution and sample elements from the estimated distribution. That way, the gradients can be computed through the stochastic sampling operation. Here, we specifically learn two models able to predict the mean and variance parameters $\mu_{\theta}(x_t, t)$ and $\Sigma_{\theta}(x_t, t)$ conditioned on $x_t$ and timestep $t$. These two estimators are trained following the ELBO objective which compares the estimated and reverse process distributions at each timestep. In practice, this is achieved by randomly sampling a timestep and optimizing the model with the corresponding loss function.
To stabilize the training, authors from [200] propose to fix $\Sigma_{\theta}(x_t, t) = \beta_t^2 \mathbf{I}$ and introduce a few simplifications in the loss. First, as the loss only involve KL divergence between Gaussians, it can be written analytically:

$$ \mathcal{L}_t = \mathbb{E}_{t,x_0,\epsilon} \left[ \frac{1}{2 \|\Sigma_{\theta}\|_2^2} \|\tilde{\mu}_t(x_t, x_0) - \mu_{\theta}(x_t, t)\|^2 \right]. \quad (7.21) $$

Another simplification follows from observing that $\tilde{\mu}_t(x_t, x_0)$ and $\mu_{\theta}(x_t, t)$ can be re-written as:

$$ \tilde{\mu}_t(x_t, x_0) = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{1 - \alpha_t}{\sqrt{1 - \alpha_t}} \epsilon_t \right), \quad (7.22) $$

$$ \mu_{\theta}(x_t, t) = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{1 - \alpha_t}{\sqrt{1 - \alpha_t}} \epsilon_{\theta}(x_t, t) \right). \quad (7.23) $$

This follows from Eqs. (7.8) and (7.11). Given this expression of $\mu_{\theta}(x_t, t)$, it is only necessary for the model to estimate $\epsilon_{\theta}(x_t, t)$, the amount of gaussian noise added to $x_{t-1}$ to produce $x_t$. This explains why diffusion models are especially well suited for denoising applications. Therefore, the loss can be further simplified as (using Eq. (7.8)):

$$ \mathcal{L}_{t}^{\text{simple}} = \mathbb{E}_{t,x_{0},\epsilon} \left[ \|\epsilon_{t} - \epsilon_{\theta}(x_{t},t)\|^{2} \right], \quad (7.24) $$

$$ = \mathbb{E}_{t,x_{0},\epsilon} \left[ \|\epsilon_{t} - \epsilon_{\theta}(\sqrt{\alpha_{t}}x_{0} + \sqrt{1-\alpha_{t}}\epsilon_{t},t)\|^{2} \right]. \quad (7.25) $$

Note that the scaling term has been omitted from the last equation as the authors from [200] obtained better results without it. The training procedure can be summarized in Algorithm 1. Finally, once the model is trained, the sampling can be done from random noise and repeatedly applying the reverse process. Using the model to estimate the noise added at each time step, we have:

$$ x_{t-1} = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{1 - \alpha_t}{\sqrt{1 - \alpha_t}} \epsilon_{\theta}(x_t, t) \right). \quad (7.26) $$

The sampling procedure is then defined in Algorithm 2. Please note that the formalism employed in this section is identical to the one used in [200]. We could have referred the reader directly to this article, but it seemed essential to recall the basic principles of diffusion models. For completeness, we also cite the excellent blogpost [302] from which we drew some inspiration for the two previous sections.

**Algorithm 1** Diffusion Training procedure

**while** not converged **do**

$x_0 \sim q(x_0)$

$t \sim \text{Uniform}(\{1, ..., T\})$

$\epsilon \sim \mathcal{N}(0, \mathbf{I})$ Take a gradient descent step on $\nabla_{\theta} \|\epsilon - \epsilon_{\theta}(\sqrt{\alpha_{t}}x_{0} + \sqrt{1-\alpha_{t}}\epsilon_{t},t)\|^{2}$

**end while**
<caption>**Algorithm 2** Diffusion Sampling Procedure</caption>

$x_T \sim \mathcal{N}(0, I)$
**for** $t = T, ..., 1$ **do**
&emsp;&emsp;$z \sim \mathcal{N}(0, I)$
&emsp;&emsp;$x_{t-1} = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{1-\alpha_t}{\sqrt{1-\alpha_t}} \epsilon_{\theta}(x_t, t) \right) + \beta_t^2 z$
**end for**
**return** $x_0$

### 7.1.3 Recent Advances with Diffusion Models

In the previous sections, we have presented the diffusion models and how they can be trained to learn complex data distributions. It was originally leveraged for image generation in [200], which samples 2D random noise and progressively generates a sensible image. Plenty of consecutive works have done the same with various improvements. In DDPM, the authors employ a denoising U-Net to estimate the noise at each time step. This U-Net is replaced with visual transformers in recent diffusion models [304]. With the above formulation, the sampling is expensive as it requires iterative application of the denoising model to the whole image. Instead, Latent Diffusion Models (LDM) [304] proposes to apply the diffusion process to the latent space to greatly reduce sampling time. The authors leverage an encoder-decoder scheme to map the image space onto the latent space and back. In addition, their formulation is well suited for latent manipulation and conditioning the generation process with additional information such as text, other images, layout, etc. Other approaches speed-up DMs with improved sampling such as strided sampling schedule [305], ODE-based sampling [306, 307], and careful variance scheduling [308]. Alternatively, some contributions reconsider the denoising diffusion process and leverage other corruption processes such as blurring [309] or masking [310]. Another approach is to leverage non-Markovian diffusion process with for instance Denoising Diffusion Implicit Models (DDIM) [311]. Similarly to LDM, [312] derives a cascaded framework to scale up the generated image size. With these iterative improvements, DMs largely outperform the state-of-the-art image generation in terms of quality. Up to now, this field was mostly dominated by GANs (e.g., [313, 314]). GANs run faster, but the gap is getting smaller and DMs overcome the GANs main issues: lack of diversity, training instabilities and mode collapse.

These techniques recently got a lot of attention out of the computer vision field with their association with Large-Language-Models (LLMs) (e.g., CLIP [291], GPT [315] or T5 [316]). These models are referred to as Visual Language Models (VLM), and combine the rich semantic latent space of LLMs with image representation to perform text-to-image generation. They are embodied by Dall-E [317], Flamingo [318] and Imagen [319], among others. These models are able to generate almost indistinguishable images (at least for the human eye) in an extremely controllable way. It is great for plenty of applications, including for creative purposes. However, it also has a large societal impact as such models can easily be misused (e.g., for deepfake generation) and are subject to questionable biases. As an example for the previous claims, Fig. 7.2 provides a few examples of real and fake images generated with various VLMs, guessing correctly which pictures are fake is quite challenging.
<figure><img src="image_85.png" /><figcaption>Figure 7.2: Examples of real and fake images generated with diffusion approaches. We encourage the reader to guess which images are real and which are fake. We provide the list of answers in a footnote¹ on the next page to prevent any confirmation bias.</figcaption></figure>
Of course, image generation is not the only task that can be handled with diffusion models. Generation of all kinds of modalities can be performed with DMs: music [320, 321], voice [322], text [323], time series [324] or graphs [325]. In addition, DMs can also be leveraged for non-generative tasks, such as image translation [326], inverse image problems [304] (application to inpainting), 3D modelling [327]. Last but not least, they can also be used for predictive tasks such as segmentation [328], and, of course, detection with DiffusionDet [74]. Thus, the next section will be dedicated to explaining the principle behind DiffusionDet.

## 7.2 DiffusionDet for Object Detection

DiffusionDet [74] is a recently proposed model for object detection. It tackles the OD task using a generative approach instead of seeing it as a regression task. The latter predicts the box coordinates from the input image while the former generates the box coordinates conditioned on the image. The difference is subtle, but not seeing the detection as a regression problem unlocks new designs. The main idea of DiffusionDet is to apply the diffusion principle to the box generation. Random boxes are first sampled, and a model is trained to refine iteratively the size and position of the boxes so that they localize the objects in the input image, this is illustrated in Fig. 7.3. Specifically, the boxes are iteratively denoised by the model. The diffusion process considered here is the same as in Denoising Diffusion Implicit Models (DDIM) [311], which as mentioned in the previous section, proposes a non-Markovian forward process that leads to the same objective as DDPM. The non-Markovian property of the novel diffusion allows for much faster denoising. DDIM sampling is then leveraged in DiffusionDet to iteratively denoise the boxes.

Specifically, the denoising part of DiffusionDet is a lightweight hybrid network, it consists of a self-attention layer (transformer-like) followed by a dynamic layer (called an Instance Interaction layer). The diffusion/detection head is finally split into two branches, one for classification and one for regression. Both branches are implemented as small MLPs. The input to the head is computed from the input images features extracted with a backbone network. The backbone is a ResNet-50 with a three level FPN attached on top. Before being fed to the detection head, object features are pooled from the entire feature map with RoI Align module. The detection head processes object features independently, but the Instance Interaction layer enables interactions between instances. The detection head is applied iteratively to refine the bounding boxes. The initial bounding boxes are sampled randomly from a normal distribution. The regression branch of the head is trained to predict the noise between the true boxes and the current boxes. After each iteration, the boxes are updated following the DDIM sampling strategy. Only a small number of iterations is required to get satisfactory boxes (the original paper provides experiments with between 1 and 8 iterations). A renewal process also replaces boxes with small confidence scores with random ones after each iteration to prevent duplicated or erroneous boxes. The dynamic layer injects features from the previous iteration into the computation of the adjusted boxes. The current time step is encoded

<footnote n="1">¹Real images are images: A, B, G, H, L and N, others have been generated with Midjourney or Dall-E.</footnote>
<figure><img src="image_86.png" /><figcaption>Figure 7.3: Illustration of DiffusionDet principle, figure taken from [74]. (a) diffusion process, (b) diffusion process for image generation with DDPM and (c) DiffusionDet principle, random boxes are denoised to locate objects in the image.</figcaption></figure>

into a time embedding using a lightweight MLP. These embeddings are then used to compute scale and shift vectors to transform the object features and condition the model. We provide a detailed architecture diagram in Fig. 7.4.

The training is done in a similar fashion as in DDIM, except for the loss function which is designed for object detection. First, a timestep is sampled randomly, then the right amount of noise is added to the ground truth boxes and the model is optimized using a classical loss function for detection (a combination of Generalized IoU, L1 loss on box coordinates and cross-entropy for the labels). As the number of predicted boxes is fixed, a set-to-set matcher is employed to build target-prediction pairs with similar functioning as in DETR [59]. The loss is then computed on the selected pairs as done in any detection framework.

<figure><img src="image_87.png" /><figcaption>Figure 7.4: DiffusionDet architecture and detailed detection head design.</figcaption></figure>
DiffusionDet has a hybrid structure, it only has one stage, yet it does not predict boxes and labels densely as common one-stage detectors. The box denoising formulation allows for replacing the first stage (i.e., the RPN) with a much more naive approach: random box sampling. The iterative refinement of the boxes is able to compensate for the poor initial positioning of the boxes. In a sense, DiffusionDet resembles two-stage detection frameworks that leverage iterative bounding box regression such as CRAFT [329], Object Detection via Multi-region model [330], or Cascade R-CNN [331]. The main difference between these models and DiffusionDet is the direct box prediction. Instead of outputting refined box location, DiffusionDet is trained to predict the shift between the current boxes and the corresponding ground truth. This does not seem significant, but it is much more adapted to the iterative regression procedure. First iterative methods propose using the same detection head repeatedly to get better and better boxes. However, the head is trained directly to output correct boxes, no matter how off they are in the first place. DiffusionDet instead conditions the model with the timestep embedding so that it knows how much noise should be removed from the boxes. With this trick, it can reuse the same head without any issues. Conversely, Cascade R-CNN makes use of decoupled heads for each iteration to account for different refinement magnitudes; however, it significantly increases the model size.

## 7.3 Few-Shot DiffusionDet

Now that we have reviewed the basic principles of diffusion models and presented DiffusionDet we can see how it can be leveraged for FSOD. In this chapter, we propose an adaptation of DiffusionDet in the few-shot setting, based on fine-tuning. Fine-tuning has become increasingly popular in FSOD throughout the last two years (see Tab. 3.1). Simple fine-tuning strategies are now competitive with elaborated attention mechanisms. Another motivation for trying a fine-tuning approach is to study techniques from the three main directions in FSOD. Chaps. 5 and 6 respectively focus on metric-learning and attention-based methods. The last kind of FSOD approach in the literature relies on fine-tuning. While such a strategy is not very innovative, this is one of the first applications of a diffusion-based approach to a few-shot predictive task. In addition to the fine-tuning strategy, we propose a transductive inference scheme to boost the performance of the fine-tuned model. However, these are only preliminary work and do not yield the expected results yet. Finally, we also investigate an attention-based extension of DiffusionDet, while promising on paper the experimental study demonstrates poor results.

### 7.3.1 Fine-tuning Strategy for Few-Shot DiffusionDet

We present in this section the fine-tuning strategy that we propose to adapt DiffusionDet to the few-shot regime:

1. Train DiffusionDet in a regular fashion on a dataset containing only examples of the base classes.

2. Once base training is done, replace the classification layer with a randomly initialized layer with as many output neurons as the number of novel classes.