<figure><img src="image_1.png" /></figure>

HAL
open science

# Object Detection in Aerial Images in Scarce Data Regimes

Pierre Le Jeune

<figure><img src="image_2.png" /></figure>

## To cite this version:

Pierre Le Jeune. Object Detection in Aerial Images in Scarce Data Regimes. Computer Science [cs]. Université Sorbonne Paris Nord, 2023. English. NNT: tel-04243994

HAL Id: tel-04243994

https://hal.science/tel-04243994

Submitted on 16 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
OBJECT DETECTION IN AERIAL
IMAGES IN SCARCE DATA REGIMES

Ph.D THESIS
PIERRE LE JEUNE

<figure><img src="image_4.png" /></figure>
<figure><img src="image_5.png" /></figure>
# OBJECT DETECTION IN AERIAL IMAGES IN SCARCE DATA REGIMES

## DÉTECTION D'OBJETS DANS DES IMAGES AÉRIENNES EN CAS DE FAIBLE SUPERVISION

### Thèse de Doctorat

Presentée et soutenue le 03-10-2023 par :

PIERRE LE JEUNE

en vue de l'obtention du grade de
DOCTEUR EN INFORMATIQUE

École doctorale Galilée (ED 146), Université Sorbonne Paris Nord
Laboratoire de Traitement et Transport de l'Information (L2TI, UR 3043)
COSE

Directrice : ANISSA MOKRAOUI

### Membres du Jury

<table><tr><td>CÉLINE HUDELOT</td><td>Professeur — CentraleSupélec</td><td>RAPPORTRICE</td></tr><tr><td>PAUL HONEINE</td><td>Professeur — Université de Rouen Normandie</td><td>RAPPORTEUR</td></tr><tr><td>EMMANUEL DELLANDREA</td><td>MCF HDR — École Centrale de Lyon</td><td>EXAMINATEUR</td></tr><tr><td>ISMAIL BEN AYED</td><td>Professeur — ETS Montréal</td><td>EXAMINATEUR</td></tr><tr><td>FANGCHEN FENG</td><td>MCF — Université Sorbonne Paris Nord</td><td>EXAMINATEUR</td></tr><tr><td>HERVÉ GUIOT</td><td>Directeur — COSE</td><td>INVITÉ</td></tr><tr><td>ANISSA MOKRAOUI</td><td>Professeur — Université Sorbonne Paris Nord</td><td>DIRECTRICE</td></tr></table>
## ABSTRACT

Most contributions on Few-Shot Object Detection (FSOD) evaluate their methods on natural images only, yet the transferability of the announced performance is not guaranteed for applications on other kinds of images. We demonstrate this with an in-depth analysis of existing FSOD methods on aerial images and observed a large performance gap compared to natural images. Small objects, more numerous in aerial images, are the cause for the apparent performance gap between natural and aerial images. As a consequence, we improve FSOD performance on small objects with a carefully designed attention mechanism. In addition, we also propose a scale-adaptive box similarity criterion, that improves the training and evaluation of FSOD methods, particularly for small objects. We also contribute to generic FSOD with two distinct approaches based on metric learning and fine-tuning. Impressive results are achieved with the fine-tuning method, which encourages tackling more complex scenarios such as Cross-Domain FSOD. We conduct preliminary experiments in this direction and obtain promising results. Finally, we address the deployment of the detection models inside COSE's systems. Detection must be done in real-time in extremely large images (more than 100 megapixels), with limited computation power. Leveraging existing optimization tools such as TensorRT, we successfully tackle this engineering challenge.

**Keywords**— Object Detection, Few-Shot Learning, Few-Shot Object Detection, Cross-Domain Adaptation, Deep Learning, Computer Vision, Intersection over Union, Attention Mechanism, Diffusion, Query-Support Alignment

## RÉSUMÉ

La plupart des contributions en Détection d'Objets *Few-Shot* (FSOD) évaluent leurs méthodes uniquement sur des images naturelles, ne garantissant pas la transférabilité de leur performance à d'autres types d'images. Nous démontrons ceci avec une analyse des méthodes FSOD existantes sur des images aériennes et observons un large écart comparé aux images naturelles. Les petits objets, plus nombreux dans les images aériennes, sont responsables de cet écart. Ainsi, nous proposons d'améliorer la détection des petits objets avec un mécanisme d'attention dédié. En plus, nous proposons un nouveau critère de similarité pour boîtes englobantes, adaptatif à la taille. Il améliore l'entraînement et l'évaluation des modèles FSOD, en particulier pour les petits objets. Nous contribuons aussi au FSOD classique avec deux approches distinctes basées sur le *metric learning* et le *fine-tuning*. Des résultats impressionnants sont obtenus avec cette dernière méthode, ce qui encourage son application à des scénarios plus complexes comme la détection *Few-Shot Cross-Domain*. Finalement, nous abordons le déploiement de modèles de détection au sein des systèmes de COSE qui doivent détecter les objets en temps réel sur de très grandes images (plus de 100 mégapixels), avec des ressources de calcul limitées.

**Mots-Clé**— Détection d'objet, Apprentissage profond, Apprentissage frugal, Adaptation au domaine, Mécanisme d'attention, Diffusion, *Intersection over Union*, Alignement query-support
REMERCIEMENTS

Je tiens tout d'abord à remercier ma directrice de thèse Anissa MOKRAOUI, Professeur à l'Université Sorbonne Paris Nord, pour son encadrement, son soutien, ses conseils avisés et sa disponibilité tout au long de cette thèse.

Je remercie également Hervé GUIOT et Olivier GUITTON de l'entreprise COSE qui ont suivi et supervisé ce projet avec attention. J'en profite pour remercier COSE de manière générale pour le financement de cette thèse. Il me faut également écrire un mot pour mes collègues dans l'entreprise qui m'ont apporté un soutien moral quotidien pendant ces trois ans. En particulier, merci à Brice, Chloé, François, Kimmeng, Lotfi, Maxime, Remi et Ugo.

Merci à l'ensemble des membres du jury de ma thèse d'avoir accepté d'évaluer mes travaux de recherche et d'avoir formulé des remarques constructives pour l'amélioration de ce manuscrit.

Je voudrais ensuite remercier Damien, Florian, Jules, Quentin, Tanguy et Thibaut pour leurs relectures attentives et leurs questions pertinentes qui m'ont grandement aidé dans la rédaction de ce manuscrit de thèse. Merci aussi pour les moments de détente passés avec vous au cours des trois dernières années qui m'ont permis de décompresser quand cela était nécessaire.

Ensuite, je voudrais remercier Hicham TALAOUBRID et Lièce CHERCHOUR, stagiaires chez COSE qui ont été d'une grande aide dans l'obtention de certains résultats, l'implémentation et le déploiement de plusieurs algorithmes.

Je remercie aussi l'ANRT pour le financement de cette thèse, le L2TI et le LabCom IRISER (ANR-21-LCV3-0004) pour leur cadre scientifique, et enfin le GENCI pour la mise à disposition de moyens de calcul de l'IDRIS au travers de l'allocation de ressources 2022-AD011013955.

Merci également à MM. BROUARD et VILQUIN, enseignants au Lycée Sivard de Beaulieu de Carentan, ils m'ont initié aux Mathématiques, à la Physique et à l'Informatique. C'est grâce à eux que j'ai cette passion pour les sciences.

Je dois aussi remercier ma famille, et en particulier ma mère de m'avoir toujours poussé et soutenu dans mes études. Sans cela, je ne serai certainement pas là où j'en suis aujourd'hui.

Enfin, je ne peux pas conclure sans remercier Marie du fond de mon cœur. Je te remercie pour tes nombreuses relectures, mais surtout pour ton soutien indéfectible au quotidien et tes encouragements souvent indispensables. Sans toi, je n'y serai pas arrivé !

À mon père.
# TABLE OF CONTENTS

Abstract i

Résumé i

Remerciements iii

Table of Contents v

List of Figures ix

List of Tables xiii

Glossary xv

List of Symbols xvi

List of Acronyms xvii

Introduction 1

1.1 Industrial Context, Motivation and Objectives 1

1.2 Overview of the thesis 6

1.3 Summary of the Contributions 9

Introduction (Français) 11

1.1 Contexte industriel, motivation et objectifs 11

1.2 Plan de la thèse 16

1.3 Résumé des contributions 20

## Part I: Literature Review on Object Detection,

Few-Shot Learning, and Few-Shot Object Detection

2 Object Detection, Few-Shot Learning and Cross-Domain Adaptation 25

2.1 Object Detection 26

2.2 Few-Shot Learning: Learning with Limited Data 44

2.3 Conclusion 59
<table><tr><td><b>3</b></td><td><b>Few-Shot Object Detection</b></td><td><b>61</b></td></tr><tr><td>3.1</td><td>Problem definition</td><td>62</td></tr><tr><td>3.2</td><td>Review of the Few-Shot Object Detection Literature</td><td>63</td></tr><tr><td>3.3</td><td>Few-Shot Object Detection on Remote Sensing Images</td><td>70</td></tr><tr><td>3.4</td><td>Extension of the Few-Shot Object Detection Setting</td><td>71</td></tr><tr><td>3.5</td><td>Dataset preparation and evaluation in the Few-Shot setting</td><td>73</td></tr><tr><td>3.6</td><td>Conclusion</td><td>76</td></tr><tr><td><b>4</b></td><td><b>Understanding the Challenges of Few-Shot Object Detection</b></td><td><b>77</b></td></tr><tr><td>4.1</td><td>Distractors in the Few-Shot Data Regime</td><td>78</td></tr><tr><td>4.2</td><td>The Increased Challenge of Aerial Images</td><td>78</td></tr><tr><td>4.3</td><td>Conclusion</td><td>83</td></tr></table>

## Part II: Improving Few-Shot Object Detection through Various Approaches

<table><tr><td><b>5</b></td><td><b>Experience Feedback about Metric Learning for FSOD</b></td><td><b>89</b></td></tr><tr><td>5.1</td><td>Motivation and Principle</td><td>90</td></tr><tr><td>5.2</td><td>Prototypical Faster R-CNN for FSOD</td><td>90</td></tr><tr><td>5.3</td><td>Performance on Artificial Data</td><td>96</td></tr><tr><td>5.4</td><td>Difficulties on Aerial Images</td><td>98</td></tr><tr><td>5.5</td><td>Insights and conclusion</td><td>102</td></tr><tr><td><b>6</b></td><td><b>Attention Framework for Fair FSOD Comparison</b></td><td><b>105</b></td></tr><tr><td>6.1</td><td>Framework Presentation and Motivation</td><td>106</td></tr><tr><td>6.2</td><td>Alignment Attention Fusion Framework</td><td>107</td></tr><tr><td>6.3</td><td>Cross-Scales Query-Support Alignment for Small FSOD</td><td>120</td></tr><tr><td>6.4</td><td>Qualitative Comparison within the AAF framework</td><td>127</td></tr><tr><td>6.5</td><td>Conclusion</td><td>129</td></tr><tr><td><b>7</b></td><td><b>Few-Shot Diffusion Detector via Fine-Tuning</b></td><td><b>133</b></td></tr><tr><td>7.1</td><td>Diffusion Probabilistic Models Principle</td><td>134</td></tr><tr><td>7.2</td><td>DiffusionDet for Object Detection</td><td>141</td></tr><tr><td>7.3</td><td>Few-Shot DiffusionDet</td><td>143</td></tr><tr><td>7.4</td><td>Application to Cross-Domain FSOD</td><td>160</td></tr><tr><td>7.5</td><td>Conclusion</td><td>166</td></tr></table>
## Part III: Rethinking Intersection Over Union

<table><tr><td><b>8</b></td><td><b>Scale-Adaptive Intersection Over Union</b></td><td><b>169</b></td></tr><tr><td>8.1</td><td>Analysis of Intersection over Union</td><td>170</td></tr><tr><td>8.2</td><td>Scale-Adaptive Intersection over Union</td><td>173</td></tr><tr><td>8.3</td><td>Scale-Adaptive Criteria Analysis</td><td>176</td></tr><tr><td>8.4</td><td>SIoU Alignment with Human Perception</td><td>185</td></tr><tr><td>8.5</td><td>Experimental Results</td><td>190</td></tr><tr><td>8.6</td><td>Conclusion</td><td>198</td></tr></table>

## Part IV: Prototyping and Industrial Application

<table><tr><td><b>9</b></td><td><b>Integration in COSE Prototypes</b></td><td><b>201</b></td></tr><tr><td>9.1</td><td>CAMELEON Aerial Intelligence System</td><td>202</td></tr><tr><td>9.2</td><td>Reducing Object Detection Model Size for Edge Computing</td><td>205</td></tr><tr><td>9.3</td><td>Inference Acceleration with TensorRT</td><td>209</td></tr><tr><td>9.4</td><td>Object Detection Pipeline for CAMELEON Prototype</td><td>212</td></tr><tr><td>9.5</td><td>Conclusion</td><td>218</td></tr><tr><td colspan="2"><b>Conclusion and Perspectives</b></td><td><b>219</b></td></tr><tr><td colspan="2">Contribution summary</td><td>219</td></tr><tr><td colspan="2">Future Research Tracks</td><td>220</td></tr><tr><td colspan="2">Remaining Industrial Challenges</td><td>222</td></tr><tr><td colspan="2"><b>Bibliography</b></td><td><b>223</b></td></tr></table>

## Appendices

<table><tr><td><b>A</b></td><td><b>Proofs of SIoU's Properties</b></td><td><b>245</b></td></tr><tr><td><b>B</b></td><td><b>Theoretical GIoU's Distribution Analysis</b></td><td><b>249</b></td></tr></table>
LIST OF FIGURES

<table><tr><td>1.1</td><td>Illustration of the three main products developed at COSE: GlobalScanner, Strike and POD Xplorer.</td><td>2</td></tr><tr><td>1.2</td><td>Description of Geospatial Intelligence (GEOINT).</td><td>3</td></tr><tr><td>2.1</td><td>Bounding boxes representations</td><td>27</td></tr><tr><td>2.2</td><td>Precision, Recall, Accuracy and IoU definitions</td><td>28</td></tr><tr><td>2.3</td><td>Precision-Recall curve computation</td><td>31</td></tr><tr><td>2.4</td><td>Object Detection timeline</td><td>32</td></tr><tr><td>2.5</td><td>Object Detection architectural choices</td><td>38</td></tr><tr><td>2.6</td><td>Overview of Pascal VOC and MS COCO datasets</td><td>46</td></tr><tr><td>2.7</td><td>Overview of DOTA and DIOR datasets</td><td>47</td></tr><tr><td>2.8</td><td>Taxonomy of Few-Shot Classification</td><td>49</td></tr><tr><td>3.1</td><td>FSOD literature timeline</td><td>63</td></tr><tr><td>3.2</td><td>Attention-based FSOD principle</td><td>68</td></tr><tr><td>3.3</td><td>Query-Support Spatial Alignment</td><td>69</td></tr><tr><td>3.4</td><td>Distinct evaluation processes for fine-tuning and other methods</td><td>76</td></tr><tr><td>4.1</td><td>Class co-occurrences in DOTA, DIOR, Pascal VOC and MS COCO</td><td>79</td></tr><tr><td>4.2</td><td>Baseline comparison between FRW, WSAAN and DANA on DOTA, DIOR and Pascal VOC datasets</td><td>81</td></tr><tr><td>4.3</td><td>Per class object size analysis on DOTA, DIOR and Pascal VOC.</td><td>82</td></tr><tr><td>4.4</td><td>Per class performance analysis for FRW on DOTA, DIOR and Pascal VOC against object size</td><td>84</td></tr><tr><td>4.5</td><td>Per class performance analysis for FRW on all datasets</td><td>85</td></tr><tr><td>5.1</td><td>Prototypical Faster R-CNN architecture</td><td>92</td></tr><tr><td>5.2</td><td>Images of MNIST-LOC dataset and classes repartition.</td><td>97</td></tr><tr><td>5.3</td><td>Prototypical Faster R-CNN results on MNIST-LOC dataset</td><td>99</td></tr><tr><td>5.4</td><td>Prototypical Faster R-CNN results on DOTA</td><td>101</td></tr><tr><td>5.5</td><td>TSNE visualization of the embedding space of PFRCNN on DOTA</td><td>102</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>6.1</td>
      <td>Attention-based Few-Shot Object Detection principle</td>
      <td>107</td>
    </tr>
    <tr>
      <td>6.2</td>
      <td>Overview of the AAF Framework</td>
      <td>108</td>
    </tr>
    <tr>
      <td>6.3</td>
      <td>Object-level augmentation techniques</td>
      <td>112</td>
    </tr>
    <tr>
      <td>6.4</td>
      <td>Support example cropping strategies</td>
      <td>114</td>
    </tr>
    <tr>
      <td>6.5</td>
      <td>Influence of the number of shots on FSOD performance on Pascal VOC</td>
      <td>117</td>
    </tr>
    <tr>
      <td>6.6</td>
      <td>Influence of the number of shots on FSOD performance on Pascal VOC per class</td>
      <td>118</td>
    </tr>
    <tr>
      <td>6.7</td>
      <td>Cross-scales Query-Support Alignment diagram</td>
      <td>122</td>
    </tr>
    <tr>
      <td>6.8</td>
      <td>XQSA small and large objects matching asymmetry.</td>
      <td>124</td>
    </tr>
    <tr>
      <td>6.9</td>
      <td>FSOD performance comparison with regular baseline on DOTA, DIOR and Pascal VOC.</td>
      <td>125</td>
    </tr>
    <tr>
      <td>6.10</td>
      <td>Support examples for base classes for qualitative assessment of attention-based methods</td>
      <td>127</td>
    </tr>
    <tr>
      <td>6.11</td>
      <td>Qualitative assessment of the detection quality of FRW, WSAAN, DANA and XQSA on DOTA with K = 10 shots on base classes.</td>
      <td>128</td>
    </tr>
    <tr>
      <td>6.12</td>
      <td>Support examples for novel classes for qualitative assessment of attention-based methods</td>
      <td>130</td>
    </tr>
    <tr>
      <td>6.13</td>
      <td>Qualitative assessment of the detection quality of FRW, WSAAN, DANA and XQSA on DOTA with K = 10 shots on novel classes.</td>
      <td>131</td>
    </tr>
    <tr>
      <td>7.1</td>
      <td>Diffusion processes illustration</td>
      <td>136</td>
    </tr>
    <tr>
      <td>7.2</td>
      <td>Examples of images generated with Diffusion Models</td>
      <td>140</td>
    </tr>
    <tr>
      <td>7.3</td>
      <td>DiffusionDet principle</td>
      <td>142</td>
    </tr>
    <tr>
      <td>7.4</td>
      <td>DiffusionDet architecture</td>
      <td>142</td>
    </tr>
    <tr>
      <td>7.5</td>
      <td>Influence of the number of shots on performance</td>
      <td>146</td>
    </tr>
    <tr>
      <td>7.6</td>
      <td>DiffusionDet initial random boxes</td>
      <td>153</td>
    </tr>
    <tr>
      <td>7.7</td>
      <td>DiffusionDet scaled proposals</td>
      <td>153</td>
    </tr>
    <tr>
      <td>7.8</td>
      <td>TSNE visualizations of transductive inference</td>
      <td>156</td>
    </tr>
    <tr>
      <td>7.9</td>
      <td>Transductive cores histograms comparison</td>
      <td>157</td>
    </tr>
    <tr>
      <td>7.10</td>
      <td>FSDiffusionDet qualitative detection results</td>
      <td>159</td>
    </tr>
    <tr>
      <td>7.11</td>
      <td>Presentation of Cross-Domain datasets</td>
      <td>161</td>
    </tr>
    <tr>
      <td>7.12</td>
      <td>COCO → X cross-domain performance spider plot</td>
      <td>163</td>
    </tr>
    <tr>
      <td>8.1</td>
      <td>IoU small object inadequacy</td>
      <td>172</td>
    </tr>
    <tr>
      <td>8.2</td>
      <td>Criteria's values against box localization error</td>
      <td>177</td>
    </tr>
    <tr>
      <td>8.3</td>
      <td>Criteria's distribution comparison (γ = -4 and κ = 16 for SIoU and GSIoU)</td>
      <td>178</td>
    </tr>
    <tr>
      <td>8.4</td>
      <td>Relaxing the constraints for criteria' distribution analysis.</td>
      <td>181</td>
    </tr>
    <tr>
      <td>8.5</td>
      <td>Influence of γ and κ on the expected value and standard deviation of SIoU.</td>
      <td>184</td>
    </tr>
    <tr>
      <td>8.6</td>
      <td>Criteria's distribution comparison (γ = -4 and κ = 16 for SIoU and GSIoU)</td>
      <td>185</td>
    </tr>
    <tr>
      <td>8.7</td>
      <td>User study rating page</td>
      <td>187</td>
    </tr>
    <tr>
      <td>8.8</td>
      <td>User study examples.</td>
      <td>187</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>8.9 Criteria' alignment with human ratings</td>
      <td>189</td>
    </tr>
    <tr>
      <td>8.10 SIoU alignment with various γ values</td>
      <td>189</td>
    </tr>
    <tr>
      <td>8.11 User study factor analysis</td>
      <td>191</td>
    </tr>
    <tr>
      <td>9.1 On-board computer architecture</td>
      <td>203</td>
    </tr>
    <tr>
      <td>9.2 Nvidia AGX Orin</td>
      <td>204</td>
    </tr>
    <tr>
      <td>9.3 Inference speed / detection accuracy tradeoffs comparison</td>
      <td>206</td>
    </tr>
    <tr>
      <td>9.4 YOLOv5 speed/accuracy tradeoff on aerial images</td>
      <td>207</td>
    </tr>
    <tr>
      <td>9.5 Illustration of a computation graph and its optimization by TensorRT</td>
      <td>210</td>
    </tr>
    <tr>
      <td>9.6 Influence of overlap on the amount of data to process</td>
      <td>213</td>
    </tr>
    <tr>
      <td>9.7 CAMELEON tiling</td>
      <td>214</td>
    </tr>
    <tr>
      <td>9.8 Qualitative detection performance of the deployed detection engine</td>
      <td>217</td>
    </tr>
    <tr>
      <td>9.9 CAMELEON user interface.</td>
      <td>217</td>
    </tr>
  </tbody>
</table>
LIST OF TABLES

<table><tr><td>2.1</td><td>Loss functions of Object Detectors</td><td>42</td></tr><tr><td>2.2</td><td>Prediction-Ground truth matching of Object Detectors</td><td>42</td></tr><tr><td>2.3</td><td>Overview of existing detection datasets</td><td>43</td></tr><tr><td>2.4</td><td>Few-Shot Learning and related tasks</td><td>55</td></tr><tr><td>3.1</td><td>FSOD literature review</td><td>64</td></tr><tr><td>3.2</td><td>Base / Novel class splits</td><td>75</td></tr><tr><td>4.1</td><td>Regular baseline performance</td><td>80</td></tr><tr><td>4.2</td><td>Performance comparison between FRW, WSAAN and DANA on DOTA, DIOR and Pascal VOC datasets</td><td>81</td></tr><tr><td>5.1</td><td>PFRCNN training tricks ablation study</td><td>96</td></tr><tr><td>5.2</td><td>PFRCNN performance on MNIST-LOC</td><td>98</td></tr><tr><td>5.3</td><td>PFRCNN performance on DOTA dataset</td><td>100</td></tr><tr><td>5.4</td><td>PFRCNN performance comparison with and without fine-tuning.</td><td>101</td></tr><tr><td>6.1</td><td>Cumulative study about augmentations</td><td>113</td></tr><tr><td>6.2</td><td>Influence of support cropping strategies on FSOD performance</td><td>115</td></tr><tr><td>6.3</td><td>FSOD performance of 5 existing methods in AAF framework on Pascal VOC</td><td>116</td></tr><tr><td>6.4</td><td>Comparison of FSOD performance on MS COCO between WSAAN and DANA</td><td>118</td></tr><tr><td>6.5</td><td>FSOD performance of AAF Framework Implementation on aerial images</td><td>119</td></tr><tr><td>6.6</td><td>mAP<sub>0.5</sub> and RmAP values for some reimplemented methods and XQSA with K = 10 shots.</td><td>120</td></tr><tr><td>6.7</td><td>XQSA components ablation study</td><td>122</td></tr><tr><td>6.8</td><td>Performance comparison between XQSA, FRW, and DANA on DOTA, DIOR, Pascal VOC and MS COCO</td><td>123</td></tr><tr><td>6.9</td><td>Performance comparison between XQSA, FRW, and DANA on DOTA, DIOR, Pascal VOC and MS COCO</td><td>124</td></tr><tr><td>7.1</td><td>FSDiffusionDet baseline compared with other FSOD methods</td><td>145</td></tr><tr><td>7.2</td><td>FSDiffusionDet performance against shots</td><td>145</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>7.3</td>
      <td>Study of the influence of the backbone pre-training</td>
      <td>147</td>
    </tr>
    <tr>
      <td>7.4</td>
      <td>Plasticity Analysis</td>
      <td>149</td>
    </tr>
    <tr>
      <td>7.5</td>
      <td>Number of proposals study</td>
      <td>150</td>
    </tr>
    <tr>
      <td>7.6</td>
      <td>Learning rate sweeping on DOTA</td>
      <td>151</td>
    </tr>
    <tr>
      <td>7.7</td>
      <td>Transductive inference against fine-tuned model</td>
      <td>157</td>
    </tr>
    <tr>
      <td>7.8</td>
      <td>Detection results of FSDiffusionDet on DOTA, DIOR, Pascal VOC and MS COCO datasets</td>
      <td>158</td>
    </tr>
    <tr>
      <td>7.9</td>
      <td>COCO → X cross-domain FSDiffusionDet performance</td>
      <td>163</td>
    </tr>
    <tr>
      <td>7.10</td>
      <td>DIOR to DOTA Cross-Domain results</td>
      <td>164</td>
    </tr>
    <tr>
      <td>7.11</td>
      <td>DOTA to DIOR Cross-Domain results</td>
      <td>165</td>
    </tr>
    <tr>
      <td>8.1</td>
      <td>Criteria statistical characteristics</td>
      <td>183</td>
    </tr>
    <tr>
      <td>8.2</td>
      <td>Criteria' correlation with human rating</td>
      <td>188</td>
    </tr>
    <tr>
      <td>8.3</td>
      <td>Influence of external variables on human rating</td>
      <td>190</td>
    </tr>
    <tr>
      <td>8.4</td>
      <td>Average criteria values for various object size</td>
      <td>192</td>
    </tr>
    <tr>
      <td>8.5</td>
      <td>Experimental comparison of SIoU with other criteria</td>
      <td>193</td>
    </tr>
    <tr>
      <td>8.6</td>
      <td>GSIoU Loss performance on DOTA, DIOR, Pascal VOC and COCO</td>
      <td>193</td>
    </tr>
    <tr>
      <td>8.7</td>
      <td>Influence of γ on the FSOD performance</td>
      <td>194</td>
    </tr>
    <tr>
      <td>8.8</td>
      <td>Influence of κ on the FSOD performance</td>
      <td>194</td>
    </tr>
    <tr>
      <td>8.9</td>
      <td>Influence of γ and κ on Pascal VOC</td>
      <td>195</td>
    </tr>
    <tr>
      <td>8.10</td>
      <td>Application of SIoU loss with various FSOD methods</td>
      <td>195</td>
    </tr>
    <tr>
      <td>8.11</td>
      <td>SIoU for regular object detection</td>
      <td>196</td>
    </tr>
    <tr>
      <td>8.12</td>
      <td>Criteria comparison with SIoU threshold during evaluation</td>
      <td>197</td>
    </tr>
    <tr>
      <td>8.13</td>
      <td>Regular object detection performance with SIoU threshold during evaluation</td>
      <td>197</td>
    </tr>
    <tr>
      <td>8.14</td>
      <td>Application on DOTA, DIOR, Pascal VOC and COCO with SIoU threshold during evaluation</td>
      <td>197</td>
    </tr>
    <tr>
      <td>9.1</td>
      <td>Performance on DOTA and DIOR for various YOLOv5 sizes</td>
      <td>207</td>
    </tr>
    <tr>
      <td>9.2</td>
      <td>Knowledge Distillation performance comparison on DOTA and DIOR</td>
      <td>209</td>
    </tr>
    <tr>
      <td>9.3</td>
      <td>CAMELEON inference profiling</td>
      <td>215</td>
    </tr>
    <tr>
      <td>9.4</td>
      <td>Engine inference speed comparison</td>
      <td>216</td>
    </tr>
    <tr>
      <td>10.1</td>
      <td>Proposed methods overall comparison</td>
      <td>221</td>
    </tr>
  </tbody>
</table>
# GLOSSARY

## List of Symbols

$f$ Backbone of a visual model (for classification or detection). Often implemented as a large CNN or Transformer model.

$\mathcal{D}_{\text{base}}$ A set containing training examples of the base classes.

$b$ A ground truth bounding box: $b = [x, y, w, h]$.

$\hat{b}$ A predicted bounding box: $\hat{b} = [\hat{x}, \hat{y}, \hat{w}, \hat{h}]$.

$c$ A semantic class, $c \in \mathcal{C}$.

$l$ A vector of classification scores, $l \in [0, 1]^c$.

$\mathcal{C}$ A set of classes.

$\mathcal{C}_{\text{base}}$ Base class set.

$\mathcal{C}_{\text{novel}}$ Novel class set.

$\mathbf{y}$ A detection label, constituted of a bounding box and a class label (or a classification score vector).

$h$ Detection or classification head, often implemented as a lightweight CNN or MLP.

$I$ An image.

$K$ Number of shots, i.e., the number of examples available for a novel class.

$\mathcal{F}(\cdot, \theta)$ General purpose model with parameters $\theta$.

$N$ Number of novel classes in the few-shot setting.

$g$ Neck of a detection model, it generally consists of a Feature Pyramidal Network.

$\mathcal{D}_{\text{novel}}$ A set containing examples of the novel classes and their annotations, used for adaptation and/or fine-tuning, a.k.a support set.

$\theta$ Parameters of a model, e.g., a neural network.

$N_p$ Number of proposals boxes in DiffusionDet.

$F_q$ Features extracted with a backbone from a query image $I_q$.

$F_s^c$ Features extracted with a backbone from a support image $I_s^c$, it represents examples from the class $c$.

$A_q$ and $A_s^c$ Aligned features in AAF framework for a query image and a support image of class $c$, respectively.
$\gamma_s$ and $\gamma_q$ Global attention operators in AAF framework for query and support images, respectively.

$H_q$ and $H_s^c$ Highlighted features in AAF framework for a query image and a support image of class c, respectively.

$\lambda_s$ and $\lambda_q$ Spatial alignment operators in AAF framework for query and support images, respectively.

$M_q^c$ Merged query features with support features for class c in AAF framework.

$\Omega$ Fusion operators in AAF framework.

$\alpha_t$ and $\beta_t$ Diffusion noise variance parameters, $\alpha_t = 1 - \beta_t$.

$\zeta$ DiffusionDet noise clamping parameter for box generation.

$\alpha$ Objectness score for an object.

$\Phi^c$ and $\Psi^c$ Prototype vectors for class c in the RPN and the detection of Prototypical Faster R-CNN, respectively.

$\xi_i$ Features extracted from Region of Interest i.

$z$ Intermediary features within a network, e.g., embedding vector or latent variable.

$\mathcal{C}$ A box similarity criterion, e.g., IoU, GIoU, or SIoU.

$\gamma$ Scale-Adaptative Intersection over Union strength parameter.

$\kappa$ Scale-Adaptative Intersection over Union scaling parameter.

List of Acronyms

k-NN k-Nearest Neighbors.

AAF Alignment Attention Fusion framework.

API Application Programming Interface.

CD Cross-Domain.

CMOS Complementary Metal-Oxide-Semiconductor.

CNN Convolutional Neural Network.

DL Deep Learning.

DM Diffusion Model.

ELBO Evidence Lower BOund.

FPGA Field-Programmable Gate Array.

FPN Feature Pyramidal Network, introduced in [1].

FSC Few-Shot Classification.

FSL Few-Shot Learning.

FSOD Few-Shot Object Detection.

G-FSOD Generalized Few-Shot Object Detection.

GEOINT Geospatial Intelligence.
<table><tr><td>GPU</td><td>Graphical Processing Unit.</td></tr><tr><td>GSD</td><td>Ground Sampling Distance.</td></tr><tr><td>IoU</td><td>Intersection over Union.</td></tr><tr><td>KD</td><td>Knowledge Distillation.</td></tr><tr><td>LDM</td><td>Latent Diffusion Model.</td></tr><tr><td>LIDAR</td><td>Light Detection and Ranging.</td></tr><tr><td>LLM</td><td>Large Language Model.</td></tr><tr><td>LR</td><td>Learning Rate.</td></tr><tr><td>MLP</td><td>Multi Layer Perceptron.</td></tr><tr><td>NMS</td><td>Non-Maximal Suppression.</td></tr><tr><td>OD</td><td>Object Detection.</td></tr><tr><td>ODE</td><td>Ordinary Differential Equation.</td></tr><tr><td>ONNX</td><td>Open Neural Network Exchange.</td></tr><tr><td>PCA</td><td>Principal Component Analysis.</td></tr><tr><td>PDF</td><td>Probability Density Function.</td></tr><tr><td>PFRCNN</td><td>Prototypical Faster R-CNN.</td></tr><tr><td>RoI Align</td><td>Region of Interest Alignment.</td></tr><tr><td>RPN</td><td>Region Proposal Network.</td></tr><tr><td>RSI</td><td>Remote Sensing Image.</td></tr><tr><td>SGD</td><td>Stochastic Gradient Descent.</td></tr><tr><td>SIoU</td><td>Scale-Adaptative Intersection over Union.</td></tr><tr><td>SMB</td><td>Small and Midsize Business.</td></tr><tr><td>SSL</td><td>Self-Supervised Learning.</td></tr><tr><td>UAV</td><td>Unmanned Aerial Vehicle.</td></tr><tr><td>XQSA</td><td>Cross-Scale Query-Support Alignment.</td></tr><tr><td>YOLO</td><td>You Only Look Once.</td></tr></table>
# INTRODUCTION

> If a machine is expected to be infallible, it cannot also be intelligent.

> - Alan Turing

As an introduction to this thesis manuscript, we present the industrial context and the motivation behind this project. First, we introduce the company COSE and the *Laboratoire de Traitement et Transport de l'Information* (L2TI) that collaborated on this CIFRE PhD project. Then, we briefly describe what object detection is and how the industrial constraints that weigh upon COSE influenced our study toward low-data regimes and few-shot learning. Next, we carry out an overview of the structure of the manuscript, with an individual summary describing each chapter. Finally, we gather the contributions that came out of this project. This includes research articles, accepted or submitted to peer review conferences and journals, as well as open-source code contributions.

## 1.1 Industrial Context, Motivation and Objectives

This PhD thesis originates from a collaboration between the L2TI laboratory from Université Sorbonne Paris Nord (USPN) and the company COSE. The L2TI was founded in 1998 and is a member of the CNRS Research Federation MathSTIC (FR 3734) which includes the *Laboratoire Analyse, Géométrie et Applications* (LAGA), UMR 7539 and the *Laboratoire d'Informatique de Paris Nord* (LIPN), UMR 7030. Two main research teams coexist in the L2TI. The Multimedia team focuses on visual information analysis and processing, while the Network team targets information transport and network questions. This thesis falls within the scope of the Multimedia team.

COSE<fnref n="1" /> is a highly innovative SMB with around 20 employees. It is a first-tier government provider in the aeronautic and defense sector. COSE was born from an INRIA start-up in the 1990s and has integrated research excellence at the heart of its industrial process. While being relatively small, COSE has multidisciplinary teams with expertise in various fields such as mechanic, electronic,

<footnote n="1">¹https://www.cose.fr/</footnote>
INTRODUCTION

<figure><img src="image_6.png" /><figcaption>(a) GlobalScanner Camera and Observation Unit.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(b) Strike stabilization arm mounted on a Gazelle helicopter.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(c) POD Xplorer next to SAFRAN's Patroller at 2019 Paris Air Show.</figcaption></figure>

Figure 1.1: Illustration of the three main products developed at COSE: GlobalScanner, Strike and POD Xplorer.

navigation, automation and embedded software. Its size gives COSE remarkable agility and cost-effectiveness in comparison to its main competitors. This competitive advantage has allowed COSE to build strong partnerships with major actors in the aeronautic and defense areas.

COSE develops, produces and supports aerial observation camera systems and onboard equipment. These products are mainly designed for military use and must therefore conform to strict quality criteria. The relationship with military forces is handled by the Directorate General of Armaments (DGA), which is one of the main clients of COSE. Among others, COSE currently relies on three products that are in use by French military forces around the world (see Fig. 1.1):

- **GlobalScanner**: a high-resolution imaging embedded system that provides real-time and georeferenced images. It consists of a high-resolution, stabilized linear sensor that can be integrated into various carriers such as helicopters, aircraft or UAVs. It comes with powerful software to operate the camera and manage image streams.

- **Strike**: a stabilization arm for helicopters to improve high-precision rifle accuracy. It improves shot accuracy and drastically reduces collateral damage.

- **POD Xplorer**: a multifunctional pod for various carriers. Its purpose is to embed various types of payloads such as optical sensors, LIDAR, scientific equipment or inertial sensors.

Recently, COSE started the CAMELEON project to replace the decades-old GlobalScanner system. Its objective is to improve GlobalScanner in every aspect. First, the linear sensor will be replaced by high-resolution CMOS matrix sensors. With up to six sensors per system, CAMELEON will be able to cover extremely large areas with high resolution. The images will also largely overlap, enabling precise 3D reconstruction of the flown-over areas, which is especially important for mission prepa-
<figure><img src="image_9.png" /><figcaption>Figure 1.2: Description of Geospatial Intelligence (GEOINT).</figcaption></figure>

ration and risk analysis. CAMELEON will also come with an improved software stack from mission planning to image analysis and visualization. This PhD project is part of this software redesign. The amount of image data acquired each second by the new system will be overwhelming for a single photo interpreter as done with GlobalScanner. Furthermore, standard communication streams will not be sufficient to send entire images in real-time. Therefore, relevant information must be extracted from the images, automatically and on edge. To this end, CAMELEON must integrate intelligent algorithms able to find relevant structures and information inside the mass of pixels acquired each second. These pieces of information are often called Geospatial Intelligence (GEOINT). They consist of evidence of human activity precisely georeferenced, with any kind of supplementary metadata (e.g., weather conditions or user annotation). Such evidence can be buildings, crop fields, vehicles, or even animals. It is illustrated in Fig. 1.2. In most cases, these are salient objects and can be detected in the images. Once an object has been localized in an image, its precise location can be derived from the carrier position, the direction of the camera and the digital elevation model used, which produces a GEOINT. The GEOINT is then enriched with relevant information about the object: what is the object? Is it dangerous? Is it moving? Even though this seems to require a human appraisal, some of these questions can be answered automatically. The main objective of this PhD project is to develop models that will be able to produce GEOINT automatically. It will need to localize objects and infer relevant metadata about them. It should drastically increase the efficiency of photo-interpreters who will then be able to manage the ever-increasing amount of data generated by aerial intelligence systems, in particular within the CAMELEON project.
Object Detection (OD) is a crucial part of creating GEOINTs. In computer vision, object detection is the task of localizing and classifying all objects visible in an image. Of course, the notion of an object needs to be defined more precisely, otherwise anything in the image can be considered of interest. A pre-defined set of semantic classes *C* is fixed so that a clear distinction can be made between objects of interest (i.e., the ones we want to detect, also called foreground objects) and background objects (i.e., those we are not interested in). Based on this distinction, the task of detecting objects can be split into two sub-tasks. 1) Localizing all the objects (foreground and background): this can be done by finding the coordinates of the center of the objects, a rectangular bounding box or even a precise segmentation mask for each object. In general, the object detection task in computer vision is associated with bounding box localization. 2) Classifying the objects localized in step 1). It consists in first filtering out background objects and then, assigning a class label *c* ∈ *C* to each foreground object. Research interest in the detection task dates back to the early 2000s when the Viola-Jones object detector [2] was first introduced. Since then, plenty of algorithms have been proposed to improve both the speed and quality of the detection. A breakthrough occurred in 2013 with the first uses of deep convolutional networks for detection, namely OverFeat [3] and R-CNN [4]. These methods paved the way for more elaborated deep-learning-based detectors. Deep-learning detectors are often referred to as learning-based approaches as they mainly rely on the *learning from data* paradigm and supervised learning. They contrast with earlier detection methods (also called traditional methods) which often build upon hand-crafted features. A thorough review of both traditional and learning-based object detectors is available in Sec. 2.1. Learning-based approaches have now established complete dominance over traditional methods in terms of detection quality while having reasonable speed performance. Therefore, most of this project will focus on learning-based algorithms.

The choice of deep-learning-based detectors may seem puzzling for COSE on-edge applications. Computing resources are limited inside the carrier. The payload must be as light as possible, so we cannot afford to embed heavy Graphical Processing Units (GPU) enabled machines, designed to run deep learning models. In addition, on-board power supplies cannot provide enough energy to run such hardware. However, light-weight, energy-efficient GPUs exist, such as the Nvidia Xavier and Orin Series, which are perfectly suited for deep-learning inference. Nevertheless, another constraint remains, images must be processed in real-time. The CAMELEON system is designed to take about 1 image every second. It is a rather low frame rate but due to the sensor size and the number of cameras, this represents a data stream of several hundreds of megapixels per second. To process such a massive amount of images every second, the detection models must be as light and efficient as possible. Fortunately, tools capable of optimizing the inference of deep learning models exist. Chap. 9 will present these tools and how they can be leveraged to build detection models fast enough for COSE's applications. This solves the issues related to the deployment and inference of such models; however, a major concern remains: how to train these object detectors?
Learning-based methods and especially deep learning models heavily rely on data to be trained. In general, the overall performance of a model highly depends on the amount and quality of annotated data available during the training. For the detection task, collecting large annotated datasets is time-consuming and expensive. In some cases, it is even impossible. In the medical domain, for instance, privacy-preserving regulations often prevent the use of personal data. For military applications, this is even harder as potential training data are classified. This is problematic for the training of data-hungry methods such as deep learning. Fortunately, there are some learning strategies much more data-efficient. These methods are usually referred to as *few-shot* or *low-shot learning*, and thorough reviews of these methods will be presented in Sec. 2.2. While there are plenty of approaches to Few-Shot Learning (FSL), all follow the same basic principle. First, learn generic knowledge about a related task (source task), second, adapt to the target task. These two training phases are referred to as base training and fine-tuning. In the case of detection, a *task* can designate a set of classes to be detected, this problem is then called Few-Shot Object Detection (FSOD). A large annotated dataset containing annotations of objects belonging to $C_{base}$ is available. The source task is to detect these objects. Then, the target task is to detect objects from the so-called *novel classes*, only provided with a limited number of annotations. Chap. 3 provides an in-depth review of existing work in this area. Generally, the target task is performed on similar images as the ones seen during base training. However, the target task can also be done with different kinds of images, e.g., the source task can be learned from natural images while the target task on aerial or medical images. This is called *Cross-Domain Adaptation*. It complexifies significantly the problem, but it is a much more realistic scenario in the industry. Collected datasets can only approximate the real data distributions. Discrepancies between the acquisition settings (i.e., camera, lights etc.) and the application settings almost always produce a performance drop. In medical imagery, this is a typical issue as different scanners will not produce exactly similar images. This prevents training models on scan collection from one hospital and deploying them in another. The military use case is another critical example. The confidentiality of the images, and the ever-changing environment and objects of interest make it difficult to build robust detection algorithms.

Given the constraints of COSE, the main objective of this project is to develop data-efficient object detection methods based on few-shot learning. We orient our research on the Few-Shot Object Detection problem, i.e., the adaptation to novel classes. While a detailed overview of the thesis will be presented in the next section, we outline here the main parts of this work. First, we conduct in Part I a thorough review of the literature about object detection, few-shot learning and finally few-shot object detection. Then, we propose three distinct FSOD approaches in Part II. This part includes experiments in the Cross-Domain setting, inside Chap. 7. Our experiments mainly focus on publicly available aerial datasets due to the lack of private datasets inside the company. These datasets contain detection annotations and will be presented in Chap. 2. Part III presents an alternative to the Intersection over Union, a bounding box similarity measure extensively employed in Object Detection. Its use for both model evaluation and training are discussed in Chap. 8. Finally, Chap. 9 provides details about the deployment of object detection models according to the needs of COSE.
## 1.2 Overview of the thesis

This section outlines the content of each chapter of this thesis. This takes the form of a small abstract per chapter. These abstracts will be repeated for convenience at the beginning of the corresponding chapters.

# Part I: Literature Review on Object Detection, Few-Shot Learning and Few-Shot Object Detection

The first part of this thesis is composed of three chapters. The two first present the literature about Object Detection, Few-Shot Learning and Few-Shot Object Detection. Then, the third chapter explores the challenges of applying Few-Shot Object Detection on aerial images and presents our first contribution: an analysis of these difficulties.

## Chap. 2: Object Detection, Few-Shot Learning and Cross-Domain Adaptation

Object Detection and Few-Shot Learning are two relevant subfields from the Computer Vision and Machine Learning fields. Both are necessary to build detection techniques able to generalize from limited data. Hence, this chapter reviews both Object Detection and Few-Shot Learning. Both problems are defined, and detailed reviews of the respective literature are conducted.

## Chap. 3: Few-Shot Object Detection

This chapter presents the task of detection in the few-shot regime and reviews the existing literature about it. Few-Shot Object Detection (FSOD) is at the crossroads of Object Detection and Few-Shot Learning, and therefore, extensively relies on these two fields explored in Chap. 2. Just as for classification, various directions are explored in the literature to tackle the detection task in the few-shot regime which will be presented in detail. Finally, this chapter focuses on the aerial image application of FSOD methods and extensions of the few-shot setting.

## Chap. 4: Understanding the Challenges of Few-Shot Object Detection

The detection task becomes extremely challenging when limited annotated data is available. In this chapter, we explore the reasons behind this difficulty. In particular, we focus on the case of aerial images for which it is even harder to apply FSOD techniques. It turns out that small objects are especially challenging for the FSOD task and are the main source of error in remote sensing images.

### Chapter's contributions:

<figure><img src="image_10.png" /></figure>

P. Le Jeune and A. Mokraoui, "Improving Few-Shot Object Detection through a Performance Analysis on Aerial and Natural Images," 2022 30th European Signal Processing Conference (EUSIPCO), Belgrade, Serbia, 2022, pp. 513-517, doi: 10.23919/EUSIPCO55093.2022.9909878.

<figure><img src="image_11.png" /></figure>

P. Le Jeune and A. Mokraoui, "Amélioration de la détection d'objets few-shot à travers une analyse de performances sur des images aériennes et naturelles." GRETSI 2022, XXVIIIème Colloque Francophone de Traitement du Signal et des Images, Nancy, France
# Part II: Improving Few-Shot Object Detection through Various Approaches

The second part of this thesis presents our main contributions to the Few-Shot Object Detection (FSOD) field. Each chapter proposes a novel approach to addressing the FSOD problem and discusses its pros and cons compared to existing methods. These contributions led to several accepted articles in international and national conferences and journals.

## Chap. 5: Experience Feedback about Metric Learning for FSOD

Prototypical Faster R-CNN (PFRCNN) is a novel approach for FSOD based on metric learning. It embeds prototypical networks inside the Faster R-CNN detection framework, specifically in place of the classification layers in the RPN and the detection head. PFRCNN is applied to synthetic images generated from the MNIST dataset and to real aerial images with DOTA dataset. The detection performance of PFRCNN is slightly disappointing but sets a first baseline on DOTA. However, the experiments conducted with PFRCNN provide relevant information about the design choices for FSOD approaches.

### Chapter's contributions:

<figure><img src="image_12.png" /></figure>

P. L. Jeune, M. Lebbah, A. Mokraoui and H. Azzag, "Experience feedback using Representation Learning for Few-Shot Object Detection on Aerial Images," 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Pasadena, CA, USA, 2021, pp. 662-667, doi: 10.1109/ICMLA52953.2021.00110.

## Chap. 6: Attention Framework for Fair FSOD Comparison

Fair comparison is extremely challenging in the Few-Shot Object Detection task as plenty of architectural choices differ from one method to another. Attention-based approaches are no exception, and it is difficult to assess which mechanisms are the most efficient for FSOD. In this chapter, we propose a highly modular framework to implement existing techniques and design new ones. It allows for fixing all hyperparameters except for the choice of the attention mechanism. Hence, a fair comparison between various mechanisms can be made. Using the framework, we also propose a novel attention mechanism specifically designed for small objects.

### Chapter's contributions:

<figure><img src="image_13.png" /></figure>

P. Le Jeune and A. Mokraoui, "A Comparative Attention Framework for Better Few-Shot Object Detection on Aerial Images", Submitted at the Elsevier Pattern Recognition journal.

<figure><img src="image_14.png" /></figure>

P. Le Jeune and A. Mokraoui, "Cross-Scale Query-Support Alignment Approach for Small Object Detection in the Few-Shot Regime", Accepted at the IEEE International Conference on Image Processing 2023 (ICIP).

## Chap. 7: Few-Shot Diffusion Detector via Fine-Tuning

Previous chapters explore few-shot object detection with metric learning and attention-based techniques. This chapter logically focuses on the last major approach for FSOD: fine-tuning. Based on DiffusionDet, a recent detection model leveraging diffusion models, we build a simple but efficient
# INTRODUCTION

fine-tuning strategy. The resulting method, called FSDiffusionDet, achieves state-of-the-art FSOD on aerial datasets and competitive performance on natural images. Extensive experimental studies explore the design choices of the fine-tuning strategy to better understand the key components required to achieve such quality. Finally, these impressive results allow considering more complex settings such as cross-domain scenarios, which are especially relevant for COSE.

**Chapter's contributions:** This chapter describes very recent work, and we plan to submit research articles to present these results.

## Part III: Rethinking Intersection Over Union

This part contains only one chapter which presents a contribution orthogonal to the approaches proposed in Part II as it questions the relevance of the Intersection over Union, a key component of object detection pipelines.

### Chap. 8: Scale-Adaptative Intersection Over Union

Intersection over Union (IoU) is not an optimal box similarity measure for evaluating and training object detectors. For evaluation, it is too strict with small objects and does not align well with human perception. For training, it provides a poor balance between small and large objects to the detriment of small ones. We propose Scale-adaptative Intersection over Union (SIoU), a parametric alternative that solves the shortcomings of IoU. We provide empirical and theoretical arguments for the superiority of SIoU through in-depth analysis of various criteria.

**Chapter's contributions:**

<figure><img src="image_15.png" /></figure>

P. Le Jeune and A. Mokraoui, "Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime", Submitted at the International Conference on Computer Vision 2023 (ICCV).

<figure><img src="image_16.png" /></figure>

P. Le Jeune and A. Mokraoui, "Extension de l'Intersection over Union pour améliorer la détection d'objets de petite taille en régime d'apprentissage few-shot", Accepted at GRETSI 2023.

## Part IV: Prototyping and Industrial Application

Finally, the last part of this thesis presents our industrial contributions. This part is crucial for COSE as it bridges the gap between research advancements and real-world applications. Therefore, the only chapter of this part discusses the engineering aspects of object detection and is not associated with any academic contribution.

### Chap. 9: Integration in COSE Prototypes

Detection models are often heavy and are not well suited for COSE's application. In this chapter, we first present in detail the CAMELEON system and its constraints. Then, we study the influence of the model size on the performance and present useful tools and tricks to accelerate the inference. Finally, we explain how the detection models are deployed inside the CAMELEON prototype and how they perform on aerial images.
# 1.3 Summary of the Contributions

## International Conference Articles

<figure><img src="image_17.png" /></figure>

P. Le Jeune, M. Lebbah, A. Mokraoui and H. Azzag, "Experience feedback using Representation Learning for Few-Shot Object Detection on Aerial Images," 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Pasadena, CA, USA, 2021, pp. 662-667, doi: 10.1109/ICMLA52953.2021.00110.

<figure><img src="image_18.png" /></figure>

P. Le Jeune and A. Mokraoui, "Improving Few-Shot Object Detection through a Performance Analysis on Aerial and Natural Images," 2022 30th European Signal Processing Conference (EUSIPCO), Belgrade, Serbia, 2022, pp. 513-517, doi: 10.23919/EUSIPCO55093.2022.9909878.

<figure><img src="image_19.png" /></figure>

P. Le Jeune and A. Mokraoui, "Cross-Scale Query-Support Alignment Approach for Small Object Detection in the Few-Shot Regime", Accepted at the IEEE International Conference on Image Processing 2023 (ICIP).

## National Conference Articles

<figure><img src="image_20.png" /></figure>

P. Le Jeune and A. Mokraoui, "Amélioration de la détection d'objets few-shot à travers une analyse de performances sur des images aériennes et naturelles." GRETSI 2022, XXVIIIème Colloque Francophone de Traitement du Signal et des Images, Nancy, France.

<figure><img src="image_21.png" /></figure>

P. Le Jeune and A. Mokraoui, "Extension de l'Intersection over Union pour améliorer la détection d'objets de petite taille en régime d'apprentissage few-shot", GRETSI 2023, XXIXème Colloque Francophone de Traitement du Signal et des Images, Grenoble, France.

## Submitted Articles

<figure><img src="image_22.png" /></figure>

P. Le Jeune and A. Mokraoui, "A Comparative Attention Framework for Better Few-Shot Object Detection on Aerial Images", Submitted at the Elsevier Pattern Recognition journal.

<figure><img src="image_23.png" /></figure>

P. Le Jeune and A. Mokraoui, "Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime", Submitted at the International Conference on Computer Vision 2023 (ICCV).

## Oral Presentations

During the PhD, I had the opportunity to give talks in various occasions listed below:

- L2TI's scientific day (Dec. 2020).

- *Prototypical Faster R-CNN for Few-Shot Object Detection on Aerial Images*, DeepLearn Summer School 2021, Las Palmas de Gran Canaria (Jul. 29, 2021).

- *Prototypical Faster R-CNN for Few-Shot Object Detection on Aerial Images* at a GDR-ISIS meeting: *Vers un apprentissage pragmatique dans un contexte de données visuelles labellisées limitées*, Paris, (Nov. 26, 2021).

- L2TI's Doctoral seminar (Mar. 2022 and Feb. 2023).

- *Few-Shot Object Detection on Aerial Images*, Seminar at ETS Montreal (Sep. 28, 2022).
## Internships Supervision

I supervised four internships over the three years of this PhD, three inside the company and one at within the L2TI:

- Conception et mise en oeuvre d'algorithmes de suivi d'objets dans des images aériennes (March-August 2021 - COSE).

- Optimisation et intégration d'algorithmes de détection d'objets dans un système embarqué (March-August 2022 - COSE).

- Self-supervised learning for Few-shot Object Detection (April-August 2022 - L2TI au travers du LabCom IRISER).

- Détection d'objets few-shot par visual transformers sur des images Aériennes (March-August 2023 - COSE and L2TI through the LabCom IRISER).

In addition to the supervision of two internships, I am actively involved inside the LabCom IRISER<fnref n="2" /> which is a joint laboratory between COSE, the L2TI and the LIPN. It was created one year after the beginning of my PhD at the instigation of my academic and industrial supervisors.

## Open-source Software

In the course of the various project I conducted during this PhD, I wrote multiple open-source Python packages that can be found on GitHub:

* Prototypical Faster R-CNN

* AAF framework

* Pycocosiou

* FSDiffusionDet

<footnote n="2">²Link to the LabCom IRISIER's website</footnote>
# INTRODUCTION (FRANÇAIS)

> Si une machine doit être infaillible, alors elle ne peut pas aussi être intelligente.

> — Alan Turing

Pour introduire ce manuscrit de thèse, le contexte industriel et les motivations de ce projet sont présentés. D'abord, sont introduits l'entreprise COSE et le Laboratoire de Traitement et Transport de l'Information (L2TI) qui ont collaboré sur cette thèse CIFRE. Ensuite, nous décrivons ce qu'est la détection d'objets dans le cadre de la vision par ordinateur et comment les contraintes industrielles liées à COSE ont orienté la thèse vers l'apprentissage frugal (dit few-shot). Dans un second temps, la structure de ce manuscrit est exposée en présentant un résumé individuel pour chaque chapitre. Enfin, une dernière partie liste les différentes contributions apportées au cours de ce projet, cela inclut des articles de recherche publiés ou soumis dans des conférences nationales et internationales.

## 1.1 Contexte industriel, motivation et objectifs

Cette thèse a pour origine la collaboration entre le laboratoire L2TI de l'Université Sorbonne Paris Nord (USPN) et la société COSE. Le L2TI a été fondé en 1998 et est un membre de la Fédération de Recherche MathSTIC du CNRS (FR 3734) qui inclut également deux laboratoires CNRS: le Laboratoire Analyse, Géométrie et Applications (LAGA), UMR 7539 et le Laboratoire d'Informatique de Paris Nord (LIPN), UMR 7030. Ces laboratoires sont tous rattachés à l'Institut Galilée. Deux équipes de recherche cohabitent dans le L2TI. D'abord, l'équipe Multimédia, qui se concentre sur le traitement et l'analyse de l'information visuelle et audio. Ensuite, l'équipe Réseaux, qui travaille sur le transport de l'information et les communications. Ce projet de thèse s'inscrit logiquement dans l'équipe Multimédia.

COSE <fnref n="1" /> est une PME innovante d'environ 20 salariés. C'est un fournisseur de rang 1 de l'état dans le secteur de l'aéronautique et de la défense. COSE est né en tant que startup de l'INRIA dans

<footnote n="1">¹https://www.cose.fr/</footnote>
les années 90 et la recherche est toujours au cœur de son processus industriel. Bien que relative-
ment petite, COSE possède des équipes pluridisciplinaires de haut niveau dans des domaines tels
que la mécanique, l'électronique, la navigation, l'automatique et les systèmes embarqués. La taille
de COSE lui confère une agilité et une efficacité remarquable comparée à ces principaux compéti-
teurs. Cet avantage permet à l'entreprise de créer des partenariats forts avec les acteurs majeurs de
l'aéronautique et de la défense.

COSE développe, produit et maintient des systèmes de renseignements aéroportés et des équipements embarqués en tout genre. Ces produits sont principalement destinés à un usage militaire et sont donc soumis à des critères de qualité stricts. La relation entre les forces armées et COSE est gérée par la Direction Générale de l'Armement (DGA) qui est de fait l'un des principaux clients de COSE. COSE a pour l'instant trois produits principaux dans sa gamme que les forces françaises utilisent pour différentes missions (voir Fig. 1.1) :

- **GlobalScanner** : un système de caméra embarquée qui produit des images haute résolution et géoréférencées en temps réel. Le système est constitué d'un capteur linéaire de très grande résolution. Ce capteur est stabilisé et intégré au sein d'une enceinte mécanique qui peut être intégrée sous différents types d'aéronefs (hélicoptère, avion, drone, etc.). Le capteur est connecté à un poste de contrôle et une suite logicielle permettant de piloter la caméra et de gérer les flux d'images.

- **Strike** : un bras de stabilisation d'arme à feu pour hélicoptère. Il améliore sensiblement la précision des tireurs et réduit les risques de dommages collatéraux.

- **POD Xplorer** : un pod multifonction pouvant être attaché en dessous de différents types de porteurs. Il permet d'embarquer simplement des charges utiles variées comme des capteurs optiques, des LIDARs, ou des équipements scientifiques.

Récemment, COSE a lancé le projet CAMELEON afin de remplacer GlobalScanner. Son objectif premier est de surpasser GlobalScanner dans tous les aspects. Premièrement, le capteur linéaire sera remplacé par un capteur matriciel CMOS de haute résolution. CAMELEON pourra embarquer jusqu'à six capteurs avec des orientations différentes afin de couvrir des très grandes zones au sol tout en conservant une grande définition. Les images ainsi acquises auront beaucoup de recouvrement afin de permettre la reconstruction 3D des zones survolées. C'est un aspect extrêmement important de la préparation de mission et la gestion des risques pour les forces armées. CAMELEON proposera également une amélioration complète du logiciel d'observation et notamment en ce qui concerne l'analyse et le traitement des images. La quantité d'images obtenues chaque seconde par le système sera trop importante pour être analysée par un seul photo-interprète. De plus, les moyens de communication standards n'ont pas un débit suffisant pour transmettre les images en temps réel. Ainsi, il est nécessaire d'extraire les informations stratégiques des images, automatiquement et à bord. CAMELEON doit donc être doté d'algorithmes intelligents et efficaces afin d'extraire les informations pertinentes en temps réel. Cette thèse s'inscrit dans la refonte logicielle de CAMELEON et tente de répondre aux contraintes du projet. Les informations extraites des images sont sou-
<figure><img src="image_24.png" /><figcaption>Figure 1.1: Illustration des trois produits phares de COSE: GlobalScanner, Strike et le POD Xplorer.</figcaption></figure>

vent appelées GEOspatial INtelligence (GEOINT). Il s'agit principalement de preuve d'activité humaine, précisément géoréférencées ainsi que de méta-données en tout genre (e.g., les conditions météorologiques ou des annotations de l'interprète). Le concept de GEOINT est illustré dans la Figure 1.2. Il peut s'agir de bâtiments, de champs, de véhicules ou même d'animaux. Dans la plupart des cas, ce sont des objets saillants qui sont visibles dans les images aériennes. Une fois qu'un objet a été localisé dans l'image, sa géolocalisation précise peut être calculée en fonction de la position du porteur, de l'angle de la caméra et du modèle numérique de terrain utilisé, cela produit ainsi un GEOINT. Il peut ensuite être enrichi avec des informations supplémentaires, pertinentes pour l'opération : quel est cet objet ? Est-ce une menace ? Est-il en mouvement ? Même si ces questions semblent requérir le jugement humain, on peut en réalité souvent y répondre automatiquement. L'objectif principal de cette thèse est de produire des modèles capables d'automatiser la création de GEOINT. Pour cela, ces modèles devront localiser les objets d'intérêt et inférer les méta-données pertinentes en lien avec ces objets. Aidés par ces outils, les photo-interprètes seront bien plus efficaces et pourront gérer des masses d'images toujours plus grandes.

La détection d'objets est une étape cruciale de la création de GEOINTs. En vision par ordinateur, la détection d'objets consiste à localiser et classifier tous les objets visibles dans une image. Bien sûr, la notion d'objet doit être définie de manière plus précise, sinon tout ce qui se trouve dans l'image peut être considéré comme étant d'intérêt. Un ensemble prédéfini de classes sémantiques C est fixé afin d'établir une distinction claire entre les objets d'intérêt (ceux que l'on souhaite détecter) et les objets de l'arrière-plan (ceux qui ne nous intéressent pas). Sur la base de cette distinction, la tâche de détection d'objets peut être divisée en deux sous-tâches. 1) Localiser tous les objets (objets d'intérêt et objets de l'arrière-plan) : cela peut être fait en trouvant les coordonnées du centre des objets, d'une boîte englobante rectangulaire ou même un masque de segmentation précis pour chaque ob-
<figure><img src="image_25.png" /><figcaption>Figure 1.2: Illustration d'un renseignement géospatial (GEOINT).</figcaption></figure>

jet. En général, la tâche de détection d'objets en vision par ordinateur est associée à la localisation par boîte englobante. 2) Classer les objets localisés à l'étape 1). Il s'agit d'abord de filtrer les objets de l'arrière-plan, puis d'attribuer une classe $c \in C$ à chaque objet d'intérêt. La recherche sur la détection d'objets a pour origine le début des années 2000, lorsque le détecteur d'objets Viola-Jones [2] a été introduit pour la première fois. Depuis, de nombreux algorithmes ont été proposés pour améliorer à la fois la vitesse et la qualité de la détection. Une avancée remarquable s'est produite en 2013 avec les premières utilisations de réseaux de neurones convolutifs pour la détection, notamment avec OverFeat [3] et R-CNN [4]. Ces méthodes ont ouvert la voie à des détecteurs de plus en plus élaborés basés sur l'apprentissage profond. Ces détecteurs reposent principalement sur le paradigme de l'apprentissage machine et notamment l'apprentissage supervisé. Ils diffèrent des méthodes de détection antérieures (appelées méthodes traditionnelles) qui s'appuient souvent sur des caractéristiques manuelles. Une revue détaillée des détecteurs d'objets traditionnels et basés sur l'apprentissage est disponible dans la section 2.1. Les approches basées sur l'apprentissage ont désormais établi une domination complète sur les méthodes traditionnelles en termes de qualité de détection, tout en offrant des temps d'exécution plus rapides. Par conséquent, la majeure partie de ce projet se concentrera sur les algorithmes basés sur l'apprentissage.

Le choix de détecteurs basés sur l'apprentissage profond peut sembler compliqué pour les applications embarquées telles que celles de COSE. Les ressources de calcul sont limitées une fois en vol. La charge utile doit être aussi légère que possible, COSE ne peut donc pas nous permettre
d'intégrer de lourdes machines dotées de cartes graphiques (GPU) conçues pour exécuter des modèles d'apprentissage profond. De plus, les alimentations embarquées ne peuvent pas fournir suffisamment d'énergie pour faire fonctionner un tel matériel. Cependant, il existe des GPU légers et économes en énergie, tels que les gammes Nvidia Xavier et Orin, qui conviennent parfaitement à des systèmes embarqués. Néanmoins, une autre contrainte subsiste : les images doivent être traitées en temps réel. Le système CAMELEON est conçu pour prendre environ une image par seconde. Il s'agit d'une fréquence d'image assez faible, mais en raison de la taille du capteur et du nombre de caméras, cela représente un flux de données de plusieurs centaines de mégapixels par seconde. Pour traiter une telle quantité d'images par seconde, les modèles de détection doivent être aussi légers et efficaces que possible. Heureusement, il existe des outils capables d'optimiser l'inférence des modèles d'apprentissage profond. Le chapitre 9 présentera ces outils et comment ils peuvent être utilisés pour construire des modèles de détection suffisamment rapides pour les applications de COSE. Cela résout les problématiques liées au déploiement et à l'inférence des modèles, cependant, une préoccupation majeure subsiste : comment entraîner ces détecteurs d'objets ?

Les méthodes basées sur l'apprentissage, et en particulier les modèles d'apprentissage profond, reposent fortement sur les données pour leur entraînement. En général, les performances globales d'un modèle dépendent de la quantité et de la qualité des données annotées disponibles lors de l'entraînement. Pour la détection, la collecte de grands ensembles de données annotées est chronophage et coûteuse. Dans certains cas, il est même impossible de rassembler de tels ensembles de données pour l'entraînement. Dans le domaine médical, par exemple, les réglementations empêchent souvent l'utilisation de données personnelles. Dans le domaine militaire, cela est encore plus difficile car les données d'entraînement sont classifiées et ne peuvent être divulguées en aucune circonstance. Cela est problématique pour l'entraînement des méthodes d'apprentissage profond, gourmandes en données. Heureusement, il existe des stratégies d'apprentissage beaucoup plus efficaces en termes de données. Ces méthodes sont généralement désignées comme l'apprentissage frugal (few-shot learning (FSL) en anglais). Une revue détaillée de ces méthodes sera présentée dans la section 2.2. Bien qu'il existe de nombreuses approches différentes pour l'apprentissage few-shot, elles suivent souvent le même principe de base. Premièrement, elles apprennent des connaissances générales sur une tâche connexe (tâche source), puis elles s'adaptent à une tâche cible. Ces deux phases d'entraînement sont appelées entraînement de base et fine-tuning. Dans le cas de la détection, une tâche désigne un ensemble de classes à détecter, et ce problème est alors appelé détection d'objets few-shot (FSOD en anglais). Un grand ensemble de données annotées contenant des annotations d'objets appartenant à Cbase est disponible. La tâche source consiste à détecter ces objets. Ensuite, la tâche cible consiste à détecter des objets des classes nouvelles, en disposant uniquement d'un nombre limité d'annotations. Le chapitre 3 fournit une revue approfondie des travaux existants dans ce domaine. Dans le cas général, la tâche cible est réalisée sur des images similaires à celles vues pendant l'entraînement de base. Cependant, la tâche cible peut aussi être réalisée avec différents types d'images. Par exemple, la tâche source peut être apprise à partir d'images naturelles tandis que la tâche cible porte sur des images aériennes ou médicales. On appelle cela l'adaptation au
## Chap. 2 : Détection d'objets, apprentissage few-shot et adaptation aux domaines

La détection d'objets et l'apprentissage few-shot sont des sous-domaines de la vision par ordinateur et de l'apprentissage automatique. Les deux sont nécessaires pour développer des techniques de détection capables de généraliser à partir de données limitées. Par conséquent, ce chapitre passe en revue à la fois la détection d'objets et l'apprentissage few-shot. Les deux problèmes sont définis et des revues détaillées des littératures respectives sont réalisées.

## Chap. 3 : Détection d'objets few-shot

Ce chapitre présente la tâche de détection dans le régime few-shot et passe en revue la littérature existante sur ce sujet. La détection d'objets few-shot (FSOD) se situe à l'intersection de la détection d'objets et de l'apprentissage few-shot, et repose donc largement sur ces deux domaines explorés dans le chapitre 2. Tout comme pour la classification, différentes approches sont explorées dans la littérature pour aborder la tâche de détection en régime few-shot. Enfin, ce chapitre se concentre sur l'application de la détection d'objets few-shot sur des images aériennes et sur les extensions du régime few-shot.

## Chap. 4 : Analyse des difficultés liées à la détection few-shot

La tâche de détection devient extrêmement difficile lorsque les données annotées sont limitées. Dans ce chapitre, les raisons derrière ces difficultés sont explorées. En particulier, nous nous concentrons sur le cas des images aériennes pour lesquelles il est encore plus difficile d'appliquer des techniques de détection few-shot. Il s'avère que les petits objets sont particulièrement difficiles à localiser en régime few-shot et sont la principale source d'erreur dans les images aériennes.

### Contributions liées à ce chapitre :

<figure><img src="image_26.png" /></figure>

* P. Le Jeune and A. Mokraoui, "Improving Few-Shot Object Detection through a Performance Analysis on Aerial and Natural Images," 2022 30th European Signal Processing Conference (EUSIPCO), Belgrade, Serbia, 2022, pp. 513-517, doi: 10.23919/EUSIPCO55093.2022.9909878.

<figure><img src="image_27.png" /></figure>

* P. Le Jeune and A. Mokraoui, "Amélioration de la détection d'objets few-shot à travers une analyse de performances sur des images aériennes et naturelles." GRETSI 2022, XXVIIIème Colloque Francophone de Traitement du Signal et des Images, Nancy, France.

## Part II : Amélioration de la détection few-shot à travers plusieurs approches

La deuxième partie de cette thèse présente nos principales contributions dans le domaine de la détection d'objets few-shot (FSOD). Chaque chapitre propose une nouvelle approche pour aborder le problème de FSOD et discute de ses avantages et inconvénients par rapport aux méthodes existantes. Ces contributions ont abouti à plusieurs articles acceptés et soumis dans des conférences et des revues internationales et nationales.

## Partie 5 : Retour d'expérience sur l'apprentissage de métrique pour FSOD

Prototypical Faster R-CNN (PFRCNN) est une approche innovante pour la détection d'objets few-shot
(FSOD) basée sur l’apprentissage de métriques. Elle intègre des réseaux de prototypes (prototypical networks) à l’intérieur de Faster R-CNN, plus précisément à la place des couches de classification dans le RPN et la tête de détection. PFRCNN est appliqué à des images synthétiques générées à partir de l’ensemble de données MNIST et à des images aériennes réelles avec le jeu de données DOTA. Les performances de détection de PFRCNN sont légèrement décevantes, mais elles établisSENT un premier point de repère sur DOTA. Les expériences menées avec PFRCNN fournissent des informations pertinentes sur les choix de conception pour les approches FSOD.

Contributions liées à ce chapitre :

<figure><img src="image_28.png" /></figure>

P. L. Jeune, M. Lebbah, A. Mokraoui and H. Azzag, "Experience feedback using Representation Learning for Few-Shot Object Detection on Aerial Images," 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Pasadena, CA, USA, 2021, pp. 662-667, doi: 10.1109/ICMLA52953.2021.00110.

Chap. 6 : Un environnement modulaire pour la détection few-shot basée sur des mécanismes d'attention

Comparator de manière équitable différents modèles est extrêmement difficile en détection d'objets *few-shot* car de nombreuses options architecturales diffèrent d'une méthode à une autre. Les approches basées sur l'attention ne font pas exception, et il est difficile d'évaluer quels mécanismes sont les plus efficaces pour le FSOD. Ce chapitre présente un environnement modulaire pour réimplémenter les techniques existantes et concevoir de nouvelles approches. Il permet de fixer tous les hyperparamètres à l'exception du mécanisme d'attention et de les comparer de manière équitable. En utilisant cet environnement, nous proposons également un nouveau mécanisme d'attention spécifiquement conçu pour les petits objets.

Contributions liées à ce chapitre :

<figure><img src="image_29.png" /></figure>

P. Le Jeune and A. Mokraoui, "A Comparative Attention Framework for Better Few-Shot Object Detection on Aerial Images", Soumis à Elsevier Pattern Recognition journal.

<figure><img src="image_30.png" /></figure>

P. Le Jeune and A. Mokraoui, "Cross-Scale Query-Support Alignment Approach for Small Object Detection in the Few-Shot Regime", Accepté à the IEEE International Conference on Image Processing 2023 (ICIP).

Chap. 7 : FSDiffusionDet: un détecteur few-shot basé sur les modèles de diffusion et une stratégie de fine-tuning

Les chapitres précédents explorent la détection d'objets *few-shot* en utilisant l'apprentissage métrique et les techniques basées sur l'attention. Ce chapitre se concentre logiquement sur la dernière grande approche pour le FSOD : le *fine-tuning*. En nous basant sur DiffusionDet, un récent modèle de détection utilisant des modèles de diffusion, nous construisons une stratégie de *fine-tuning* simple et efficace, baptisée FSDiffusionDet. FSDiffusionDet surpasse état de l'art en FSOD sur des jeux de données aériens et obtient des performances compétitives sur les images naturelles. Des études expérimentales approfondies explorent les choix de conception de la stratégie de fine-
tuning afin de mieux comprendre les composantes clés nécessaires pour atteindre une telle qualité. Enfin, ces résultats impressionnants permettent de considérer des scénarios plus complexes comme l'adaptation à de nouveaux domaines, ce qui est particulièrement pertinent pour COSE.

***Contributions liées à ce chapitre :*** Ce chapitre décrit des travaux très récents et nous planifions de soumettre des articles de recherche qui les présenterons.

## Part III: Repenser l'Intersection over Union

Cette partie ne contient qu'un seul chapitre qui présente une contribution indépendante des approches proposées dans la partie précédente. Ce chapitre remet en question la pertinence de l'Intersection sur Union (IoU), un élément clé des modèles de détection d'objets.

### Partie 8: Intersection over Union adaptable à la taille des objets

L'Intersection sur Union (IoU) n'est pas une mesure de similarité de boîte englobante optimale pour l'évaluation et l'entraînement des détecteurs d'objets. Pour l'évaluation, elle est trop stricte avec les petits objets et ne correspond pas bien à la perception humaine. Pour l'entraînement, elle crée un déséquilibre entre les petits et grands objets souvent au détriment des petits. Nous proposons l'Intersection sur Union adaptative à l'échelle (appelée SIoU), une alternative paramétrable qui résout les lacunes de l'IoU. Des arguments empiriques et théoriques sont avancés pour démontrer la supériorité de la SIoU grâce à une analyse approfondie de celle-ci et d'autres critères existants.

### Contributions liées à ce chapitre :

<figure><img src="image_31.png" /></figure>

P. Le Jeune and A. Mokraoui, "Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime", Soumis à International Conference on Computer Vision 2023 (ICCV).

<figure><img src="image_32.png" /></figure>

P. Le Jeune and A. Mokraoui, "Extension de l'Intersection over Union pour améliorer la détection d'objets de petite taille en régime d'apprentissage few-shot", GRETSI 2023, XXIXème Colloque Francophone de Traitement du Signal et des Images, Grenoble, France.

## Partie IV: Prototypage et déploiement industriel

Enfin, la dernière partie de cette thèse présente nos contributions industrielles. Cette partie est cruciale pour COSE car elle comble l'écart entre les avancées de la recherche et les applications du monde réel. Par conséquent, le seul chapitre de cette partie aborde les aspects techniques de la détection d'objets et n'est associé à aucune contribution académique.

## Chap. 9: Intégration dans les prototypes de COSE

Les modèles de détection sont souvent lourds et ne conviennent pas bien à l'application de COSE. Ce chapitre présente d'abord en détail le système CAMELEON et ses contraintes. Ensuite, nous étudions l'influence de la taille du modèle sur ses performances et présentons des outils et des astuces utiles pour accélérer l'inférence. Enfin, nous expliquons comment les modèles de détection sont déployés à l'intérieur du prototype CAMELEON et comment ils se comportent sur des images aériennes.
# 1.3 Résumé des contributions

## Articles de conférences internationales

<figure><img src="image_33.png" /></figure>

P. L. Jeune, M. Lebbah, A. Mokraoui and H. Azzag, "Experience feedback using Representation Learning for Few-Shot Object Detection on Aerial Images," 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Pasadena, CA, USA, 2021, pp. 662-667, doi: 10.1109/ICMLA52953.2021.00110.

<figure><img src="image_34.png" /></figure>

P. Le Jeune and A. Mokraoui, "Improving Few-Shot Object Detection through a Performance Analysis on Aerial and Natural Images," 2022 30th European Signal Processing Conference (EUSIPCO), Belgrade, Serbia, 2022, pp. 513-517, doi: 10.23919/EUSIPCO55093.2022.9909878.

<figure><img src="image_35.png" /></figure>

P. Le Jeune and A. Mokraoui, "Cross-Scale Query-Support Alignment Approach for Small Object Detection in the Few-Shot Regime", Accepté à the IEEE International Conference on Image Processing 2023 (ICIP).

## Articles de conférences nationales

<figure><img src="image_36.png" /></figure>

P. Le Jeune and A. Mokraoui, "Amélioration de la détection d'objets few-shot à travers une analyse de performances sur des images aériennes et naturelles." GRETSI 2022, XXVIIIème Colloque Francophone de Traitement du Signal et des Images, Nancy, France.

<figure><img src="image_37.png" /></figure>

P. Le Jeune and A. Mokraoui, "Extension de l'Intersection over Union pour améliorer la détection d'objets de petite taille en régime d'apprentissage few-shot", GRETSI 2023, XXIXème Colloque Francophone de Traitement du Signal et des Images, Grenoble, France.

## Articles soumis

<figure><img src="image_38.png" /></figure>

P. Le Jeune and A. Mokraoui, "A Comparative Attention Framework for Better Few-Shot Object Detection on Aerial Images", Soumis à Elsevier Pattern Recognition journal.

<figure><img src="image_39.png" /></figure>

P. Le Jeune and A. Mokraoui, "Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime", Soumis à International Conference on Computer Vision 2023 (ICCV).

## Présentations orales

Au cours de cette thèse, j'ai eu l'opportunité de donner plusieurs présentations orales :

- Journée scientifique du L2TI (Déc. 2020).

- *Prototypical Faster R-CNN for Few-Shot Object Detection on Aerial Images* DeepLearn Summer School 2021, Las Palmas de Gran Canaria (29 Juil. 2021).

- Prototypical Faster R-CNN for Few-Shot Object Detection on Aerial Images à la journée du GDR-ISIS : *Vers un apprentissage pragmatique dans un contexte de données visuelles labellisées limitées*, Paris, (26 Nov. 2021).

- Séminaires des doctorants (Mar. 2022 and Fév. 2023).

- Séminaire à l'ETS Montreal: *Few-Shot Object Detection on Aerial Images* (28 Sep. 2022).
## Supervision de stages

J'ai supervisé 4 stages au cours de ces trois ans de thèse, 3 au sein de l'entreprise et un au L2TI :

- Conception et mise en oeuvre d'algorithmes de suivi d'objets dans des images aériennes (Mars-Août 2021 – COSE).

- Optimisation et intégration d'algorithmes de détection d'objets dans un système embarqué (Mars-Août 2022 – COSE).

- Apprentissage auto-supervisée pour la détection d'objets few-shot (Avril-Août 2022 – L2TI au travers du LabCom IRISER).

- Détection d'objets few-shot par visual transformers sur des images aériennes (Mars-Août 2023 – COSE et L2TI via le LabCom IRISER).

En plus des supervisions de deux stages, je suis activement impliqué dans le LabCom IRISER<fnref n="2" />, un laboratoire commun entre COSE, le L2TI et le LIPN. Ce laboratoire commun a été créé un an environ après le début de ma thèse sous l'impulsion de mes superviseurs académique et industriel.

## Logiciels libres

Au travers des différents projets qui ont constitués cette thèse, j'ai développé plusieurs package Python qui se trouvent en accès libre sur GitHub :

- Prototypical Faster R-CNN

- AAF framework

- PycocosIou

- FSDiffusionDet

<footnote n="2">²Lien vers le site du LabCom IRISER</footnote>
# INTRODUCTION
# PART I

## LITERATURE REVIEW ON OBJECT DETECTION, FEW-SHOT LEARNING, AND FEW-SHOT OBJECT DETECTION
CHAPTER 2

# OBJECT DETECTION, FEW-SHOT LEARNING AND CROSS-DOMAIN ADAPTATION

## ABSTRACT

Object Detection and Few-Shot Learning are two relevant challenges from the Computer Vision and Machine Learning fields. Both are necessary to build detection techniques able to generalize from limited data. Hence, this chapter reviews both Object Detection and Few-Shot Learning. The two problems are defined and detailed reviews of the respective literature are conducted.

## CONTENTS

<table><tr><td>2.1 Object Detection</td><td>26</td></tr><tr><td>2.1.1 Problem Definition</td><td>26</td></tr><tr><td>2.1.2 Evaluation of Object Detectors</td><td>28</td></tr><tr><td>2.1.3 Literature review about Object Detection</td><td>32</td></tr><tr><td>2.1.4 Datasets Presentation.</td><td>43</td></tr><tr><td>2.2 Few-Shot Learning: Learning with Limited Data</td><td>44</td></tr><tr><td>2.2.1 Few-shot Classification</td><td>45</td></tr><tr><td>2.2.2 Cross Domain Adaptation</td><td>56</td></tr><tr><td>2.3 Conclusion</td><td>59</td></tr></table>

As briefly presented in the introduction, this PhD project lies at the intersection of three sub-domains of Machine Learning: Object Detection (OD), Few-Shot Learning (FSL), and Cross-Domain Adaptation (CDA). In this chapter, we define more precisely what these three fields are and outline the main existing contributions in the literature. We start by introducing the main computer vision problems these fields address and the related notations adopted in this manuscript. Then, we provide a review of existing works in each area, and finally, we link them with the industrial needs of COSE.
## 2.1 Object Detection

### 2.1.1 Problem Definition

Object detection consists in localizing and classifying all objects of interest visible in an image *I*. There are multiple terms to explain this statement. First, the notion of the object of interest is defined according to a fixed set of semantic classes *C*. The objects of interest are the ones that belong to one class *c* ∈ *C*. Of course, one can question the belonging of an object to a class. A class could be ambiguous for multiple reasons. Given the quality of the image, it can be difficult to determine the class of the object depicted. For instance, in a satellite photograph, a car could be so small that it cannot be perceived by a human observer. Another issue is the slackness of our concept of objects, one word can refer to multiple objects (e.g., spring, game, or chest), and our vocabulary is organized hierarchically (e.g., the class *vehicle* contains many classes including *car* and *truck*). One could go even further by questioning the very concept of objects in our mind (see for instance exemplar-based vs. prototype-based concept theories), but it would have more to do with cognitive science than computer vision. Generally, for object detection, these complications are not considered. One object can only belong to one class and whether it belongs to the class or not falls under the common sense of the observer. Most of the time, this is established with the ground truth annotations of human experts. This explanation is rather obvious, but keep in mind that this is a simplification, this will be useful when the notion of an object gets blurrier in the case of few-shot learning and few-shot object detection.

The detection task consists in finding all occurrences of the objects of interest in the images, i.e., the image coordinates of each object. This can be done in several ways, by locating a single pixel inside each object, by drawing a rectangular bounding box, or by computing a precise mask around it. The former setting is barely used as it can be quite ambiguous, all points in an object are equivalent and no supplementary information (size and shape) can be inferred from this representation. Traditionally, object detection leverages bounding boxes to localize the objects, and precise masks are reserved for the Instance Segmentation task. A bounding box is generally determined by four coordinates, it can be the coordinates of two diagonally opposed points on the box or the coordinates of one point plus the width and height of the box, see Fig. 2.1 for more details about boxes representations. In the following, we adopt the latter definition of a bounding box: the first two coordinates denote the *x* and *y* image coordinates of the top-left corner of the box, while the last two represent the width *w* and height *h* of the box. A bounding box *b* is then denoted as follows:

$$b = [x, y, w, h]^T. \qquad (2.1)$$

As the goal of object detection is to localize and classify the objects of interest, each bounding box must be associated with a class *c* ∈ *C*. Therefore, we define the detection label as a tuple of a bounding box and a class label:

$$y = (b, c). \qquad (2.2)$$
<figure><img src="image_40.png" /><figcaption>Figure 2.1: Three different box representations: top-left and bottom-right points, top-left point, width and height, and center point width and height. Many more representations exist but they will not be presented here. In this manuscript, the second representation, top-left point width and height, will be used exclusively.</figcaption></figure>

An image may contain more than one object and therefore, each image *I* is associated with a set of detection labels $\mathcal{Y} = \{y_i\}_{i=1}^{N_I}$, where $N_I$ is the number of objects in image *I*. Hence, solving an object detection task is to find a detector model $\mathcal{F}(\cdot, \theta)$ with parameters $\theta$ able to output a set of predicted labels given an input image *I*:

$$ \mathcal{F}(I, \theta) = \hat{\mathcal{Y}} = \{\hat{y}_i\}_{i=1}^{M_I}. \qquad (2.3) $$

We employ here a slight abuse of notation by calling the output of a detector $\mathcal{Y}$. Indeed, a detector predicts a classification score $l^c$ for each class $c \in \mathcal{C}$. Therefore, $\hat{\mathcal{Y}} = (\hat{b}, \{l^c\}_{c \in \mathcal{C}})$ and $\hat{c} = \arg\max_{c \in \mathcal{C}} l^c$. For convenience, we denote $l \in [0, 1]^{|\mathcal{C}|}$ the vector of classification scores. Hatted symbols represent the model's outputs. Note that the number of detections found by the model $M_I$ may not equal the number of objects present in the image as the detector can either miss some objects or output false detections. The proximity between the predicted labels $\hat{\mathcal{Y}}$ and the ground truth labels $\mathcal{Y}$ determines the performance of the model $\mathcal{F}(\cdot, \theta)$. Hence, finding an optimal detection model is to find a set of optimal parameters, which minimizes the distance between predicted and ground truth labels:

$$ \theta^* = \arg\min_{\theta} d(\hat{\mathcal{Y}}_{\theta}, \mathcal{Y}), \qquad (2.4) $$

where $d$ is a distance measure between $\hat{\mathcal{Y}}_\theta$, the labels predicted by $\mathcal{F}(\cdot, \theta)$ and the ground truth labels $\mathcal{Y}$. Of course, there are plenty of valid approaches to measure the proximity between two sets of detection labels, some will be introduced in Sec. 2.1.2.

For COSE, detecting the objects of interest in an image is a crucial step. This step is necessary for the creation of GEOINTs. From the bounding boxes coordinates in the image, and the carrier position (latitude, longitude, and altitude) and attitude (pitch, roll, and yaw), one can determine the precise locations of the objects on Earth. These computations also involve camera properties and orientations, but they will not be addressed in this manuscript.
<figure><img src="image_41.png" /><figcaption>Figure 2.2: Definition of the Precision, Recall and Accuracy metrics (a) as well as the box similarity criterion Intersection over Union (b).</figcaption></figure>

## 2.1.2 Evaluation of Object Detectors

Before jumping into the Object Detection literature, let's introduce the most commonly used metrics employed to assess the quality of the detection models. As mentioned in the previous section assessing the detection performance of a model consists in comparing the set of predicted detection labels $\hat{\mathcal{Y}}$ with the set of ground truth labels $\mathcal{Y}$ (typically made by a human observer). In the previous section, we defined the set of detection labels over an image. However, to better assess the generalization capabilities of the detectors, their evaluation is always conducted on a relatively large test set of images. Therefore, we extend the notation $\hat{\mathcal{Y}}$ and $\mathcal{Y}$ as the sets of predicted and ground truth labels (respectively) over all test images.

### 2.1.2.1 Average Precision and mean Average Precision

The most commonly used metrics for Object Detection are the Average Precision (AP) and its extension in the multiclass setting, the mean Average Precision (mAP). The AP is formally defined as the area under the precision-recall curve:

$$AP = \int_{0}^{1} \text{Prec}(r) dr, \qquad (2.5)$$

where Prec denotes the Precision and r, the Recall. The Precision and Recall are two well-known metrics often used to evaluate classification problems. They are defined respectively as the ratio of true positive labels over the positive predicted labels and the ratio of the positive labels over the positive true labels. Figure 2.2a clearly illustrates these definitions with a classification confusion matrix. Note that the notion of True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN) introduced in the figure will be leveraged throughout the manuscript.

For now, forget about the classification part of the detection and consider only the localization problem. How can we define the four corners of the confusion matrix for bounding boxes? An approach is to consider a detection as TP if the predicted bounding box has the same coordinates as one true detection label. However, it is extremely challenging to get a perfect positioning of
the predicted boxes. First, the detector generally leverages regression techniques to predict box coordinates and outputs continuous box coordinates. This is incompatible with the annotated box coordinates, which are usually discrete. Rounding errors can cause TP to become FP. Then, from an application viewpoint, pixel-perfect bounding boxes are not necessary. Therefore, it is generally admitted that a true positive detection is a box close enough to a ground truth box. The similarity between two bounding boxes is almost always computed with the Intersection over Union (IoU). Then, TPs are the boxes that have an IoU with a true box above a fixed threshold (typically 0.5). However, the IoU may not be an optimal criterion in certain cases as we will discuss in Chap. 8.

The Intersection over Union, also known as the Jaccard index, is a well-known similarity measure between sets A and B:

$$IoU(A, B) = \frac{|A \cap B|}{|A \cup B|} \qquad (2.6)$$

Besides its application in statistics, the IoU is widely used in computer vision to assess the quality of visual tasks such as detection and segmentation. The IoU can compute how close two sets of pixels are and thus gives a similarity measure between ground truth and the model prediction. Here, we focus on the detection task, therefore the IoU can be written in terms of coordinates of the boxes $b_1 = [x_1, y_1, w_1, h_1]^T$ and $b_2 = [x_2, y_2, w_2, h_2]^T$:

$$A_{\text{inter}} = \max(0, \max(x_1, x_2) - \min(x_1 + w_1, x_2 + w_2)) \qquad (2.7) \\ * \max(0, \max(y_1, y_2) - \min(y_1 + h_1, y_2 + h_2)),$$

$$IoU(b_1, b_2) = \frac{A_{\text{inter}}}{w_1h_1 + w_2h_2 - A_{\text{inter}}} \qquad (2.8)$$

The IoU is, therefore, a crucial part of the evaluation protocol of the object detectors as it conditions which predicted bounding boxes are TP, and which are FP. The IoU threshold limit determines how close to the ground truth the predicted boxes must be to be considered TP. In the Pascal VOC detection challenge [5], this threshold was set to 0.5, this has been the gold standard for a few years. However, it changed when the more challenging Microsoft COCO dataset [6] was proposed. The authors of the COCO challenge compute the AP at several thresholds (ranging from 0.5 to 0.95) and average the values. While this is the current standard for object detection evaluation, the few-shot object detection literature still uses the former Pascal VOC AP as it is an easier metric. Hence, we will use both of these metrics in the manuscript. We will denote them as AP0.5 and AP0.5:0.95 respectively.

Before computing the precision-recall curve, the predicted labels must be ranked by confidence scores. It is usually possible to derive a confidence score along with the bounding box coordinates and labels from a detection model. This can be for instance the highest class probability score. Once the detections are ranked according to confidence scores, one can compute the running precision Preck and recall rk by taking only the top-k bounding boxes. Then, it is possible to plot the precision as a function of the recall by plotting the points (rk, Preck), for $1 \le k \le |\hat{Y}|$. This generally gives
a zig-zag shaped curve Prec(r) as visible in Fig. 2.3. Therefore, it may not be easy to compute the area under the curve, i.e., the AP. A few tricks were introduced in [7], and later popularized with the Pascal VOC challenge [5]. They consist in taking an interpolated precision-recall curve:

$$ Prec_{interpolated}(r) = \max_{\tilde{r} \ge r} Prec(\tilde{r}), \qquad (2.9) $$

and discretize the area computation over 11 equally spaced points along the recall axis. Hence, the original AP definition becomes:

$$ AP = \frac{1}{11} \sum_{i=0}^{10} Prec_{interpolated}(0.1 \times i). \qquad (2.10) $$

So far, we only discuss the evaluation without taking into account the class of the bounding boxes. In order to take this into account, the AP is computed independently for each class and noted AP<sub>c</sub>. The predicted boxes are now considered true positive only if they have a sufficient IoU with a ground truth box and if they have the same class. The mean Average Precision is defined as follows:

$$ mAP = \frac{1}{|C|} \sum_{c \in C} AP_c. \qquad (2.11) $$

The mAP is largely the most employed metric for object detection and most of our analysis will be based on it. However, there exist complementary metrics that are worth presenting here.

### 2.1.2.2 mean Average Precision per object size

The AP and mAP can be computed only on certain object sizes. The principle is simple, simply filter the sets of predicted and true labels to keep boxes of a certain size before the AP computation. This distinction was introduced in the COCO challenge [6], with three different object sizes:

- **Small**: boxes with an area $a = wh$ smaller than $a \le 32^2$ pixels.

- **Medium**: boxes with $32^2 < a \le 96^2$.

- **Large**: boxes with $a > 96^2$.

This distinction is extremely relevant for COSE applications as the objects of interest in aerial images are often small and because most detectors struggle to detect them. Even worse, this issue is reinforced in the few-shot regime as we will see in the following chapters. Therefore, mAP<sup>S</sup>, mAP<sup>M</sup> and mAP<sup>L</sup> will be extensively employed in our analysis.

### 2.1.2.3 Average Recall

Similar to the average precision, the Average Recall (AR) is computed as the area under the Recall-IoU curve:

$$ AR = 2 \int_{0.5}^{1} \text{Recall}(v) dv, \qquad (2.12) $$
<table><thead><tr><td>Rank</td><td>Correct?</td><td>Precision</td><td>Recall</td></tr></thead><tbody><tr><td>1</td><td>True</td><td>1.0</td><td>0.2</td></tr><tr><td>2</td><td>True</td><td>1.0</td><td>0.4</td></tr><tr><td>3</td><td>False</td><td>0.67</td><td>0.4</td></tr><tr><td>4</td><td>False</td><td>0.5</td><td>0.4</td></tr><tr><td>5</td><td>False</td><td>0.4</td><td>0.4</td></tr><tr><td>6</td><td>True</td><td>0.5</td><td>0.6</td></tr><tr><td>7</td><td>True</td><td>0.57</td><td>0.8</td></tr><tr><td>8</td><td>False</td><td>0.5</td><td>0.8</td></tr><tr><td>9</td><td>False</td><td>0.44</td><td>0.8</td></tr><tr><td>10</td><td>True</td><td>0.5</td><td>1.0</td></tr></tbody></table>

<figure><img src="image_42.png" /><figcaption>Figure 2.3: Example of a Precision-Recall curve and its interpolated variant.</figcaption></figure>

where $v$ denotes the IoU threshold (ranging from 0.5 to 1) for the recall computation. Similarly to the AP for different object sizes, the average recall can be declined according to the maximum number of detection allowed. Basically, this controls the size of $\hat{Y}$, the more detection the model can output, the less likely it is to miss an object of interest. Although it can be a critical metric in some applications (e.g., lesion detection), this is not essential for COSE's applications.

#### 2.1.2.4 Average Precision shortcomings and Alternative

Even though AP is widely used in the computer vision community, it has several shortcomings. Similar values of AP can be obtained from very different precision-recall curves, hiding different detectors' behavior. The ranking of the confidence scores makes the AP sensitive to the prediction confidence. Finally, the interpolation trick from [7] may cause large errors when the number of instances of the class is small. These drawbacks were highlighted in [8], which proposes an alternative metric, the Localization-Recall-Precision (LRP). This metric is an aggregation of three metrics based on the box regression error, the precision, and the recall, under a certain confidence threshold. Hence, LRP fixes some of the AP's shortcomings.

More recently, [9] also outlined two detection issues that are not spotted by AP, namely *spatial hedging* and *category hedging*. Spatial hedging comes from the fact that low-confidence duplicates (slightly perturbed spatially) of a box do not degrade the AP value, instead having a lot of these duplicates generally improves the AP. However, these duplicates are mostly burdensome from an application viewpoint. The authors even highlight some tricks in recent object detectors that boost AP while increasing the number of duplicates. Category hedging comes from duplicated boxes with different classes. Consequently, the authors proposed two novel metrics to specifically assess spatial and category hedging: Duplicate Confusion (DC) and Naming Error (NE). Note that LRP partly assesses spatial hedging as well.
### 2.1.3 Literature review about Object Detection

In this section, we review the Object Detection literature but only present the most ground-breaking works. For an exhaustive review of object detection, we defer the reader to two popular surveys [10, 11]. This section is divided into three parts, traditional object detection, CNN-based OD and Transformer based OD. These correspond to three phases in the development of object detection techniques. This is highlighted by the timeline in Fig. 2.4, which summarizes the history of the object detection field.

<figure><img src="image_43.png" /><figcaption>Figure 2.4: Timeline of the Object Detection literature, it includes some of the most relevant works in the field of Object Detection. Papers marked with a flag are the most ground-breaking works, some will be detailed in Sec. 2.1.3</figcaption></figure>

#### 2.1.3.1 Traditional Object Detection Approaches

The very beginnings of the object detection field date back to the early 1990s. It began with an easier one-class problem: face detection. Of course, there were prior works addressing this task, but they mainly focused on face recognition and not detection. The difference is slight, the recognition task only asks whether there is a face or not in an image. This field gained substantial interest over the 1970s and 1980s with seminal works such as [12, 13, 14, 15]. However, it was only in 1991 that the detection task was first tackled by EigenFaces [16]. In this work, the authors perform a Principal Component Analysis (PCA) on a set of face images. The PCA returns a set of eigenvectors (denoted as *EigenFaces*) that span the face space. Applying the EigenFaces on a sliding window over the images allows creating *faceness* maps and therefore localizing faces. Following this pioneer contribution, many face detectors were proposed in the 1990s, for instance [17, 18, 19, 20]. We called this section "Traditional Object Detection Approaches" in contrast with the two following sections that are deep learning-based approaches. However, note that a significant proportion of the methods developed during the 1990s actually leverage neural networks. In EigenFaces [16], for instance, the authors discuss an implementation of their system using Multi-Layer Perceptron (MLP) for fast parallel computation. Similarly, [19, 20] exploit and train neural networks for the task of face detection. We will see in the next section that plenty of these ideas will be re-used 20 years later
by current deep-learning-based detectors. The missing pieces in these early days of object detection were large annotated datasets and dedicated hardware such as Graphical Processing Units (GPUs).

In the late 1990s, the object detection task as we know it today with multiple classes of interest was still far off, but some research groups began to apply it to objects other than faces. As an example, [21] introduced a general probabilistic model for the object of interest used for the visual search of faces and hands. Later, [22] stepped aside from the PCA-like object representations and proposed generic learnable features based on Haar wavelets transform. This was successfully applied to pedestrian detection. An extension with slightly more elaborated features proposed the Viola-Jones detector [2]. But it mainly provided several tricks for fast computation, achieving robust real-time detection.

In the 2000s, plenty of works shared the same strategy as the Viola-Jones detector: learning a set of elaborated features and classifying regions in the input images by comparison with the set of features. Improvements were made using more and more complex feature sets [23, 24] and the popularization of Support Vector Machines (SVM) classifiers[25]. This strategy led to the well-known Histogram of Orient Gradient (HOG) [26], which was first applied to pedestrian detection. However, this method was one of the first to tackle multi-class object detection in the first Pascal VOC challenge [5] in 2005.

The Pascal VOC challenge quickly became a reference in the Object Detection field, with increasing difficulty over the years (more classes and more images). The winners of the following editions 2006, 2007 and 2008 all took inspiration from the HOG features. In particular, [27] employs several tricks to improve the detection quality based on HOG features. Among those, Deformable Part Models (DPM), i.e., representing each object as a set of its parts provide significant improvements. It also leverages pyramid features and hard examples mining which are common components of recent object detectors. DPM were then refined with for instance the Grammar Models [28] and Star Models [28].

### 2.1.3.2 Object Detection in the Deep Learning Era

While there were a few attempts to solve object detection with neural networks during the 1990s, all were limited to single-class problems and lagged behind state-of-the-art in terms of detection quality and speed. However, this changed with the popularization of fast parallel processing units (GPUs) and the creation of large image datasets. In 2012, AlexNet [29] was introduced for image classification with deep convolutional networks (CNNs). Since AlexNet, deep learning was successfully applied to most tasks in computer vision including Object Detection.

From the beginning of the deep object detection era, two schools of thought emerged: one-stage detectors and two-stage detectors. As two-stage detectors were proposed first, we will present them first here.
## Two-Stage Detectors

Regions with CNN features or R-CNN [4] is one of the first attempts to tackle the task of Object Detection with CNNs. This marks a significant performance improvement over the previous methods (about 20% mAP improvement over the best DPMs on the 2010 Pascal VOC challenge). The idea behind R-CNN is to leverage the classification power of CNNs such as AlexNet for the detection task. It first employs Selective Search [30], a class-agnostic object locator, to generate region proposals. For each region proposal, the corresponding part in the input image is cropped and fed to a CNN pre-trained on ImageNet. The CNN outputs high-dimensional feature vectors which are then classified by an SVM for each class. The CNN is fine-tuned for the detection task by replacing its final classification layer with N+1 class logits (one additional class for the background) and training with a Cross-Entropy (CE) loss function. Proposal regions with an IoU of 0.5 with a ground truth box are selected as positive proposals and the model is trained to classify these regions with the label of their corresponding ground truth. The other proposals (denoted negative proposals) are selected as background examples. The authors train classification SVMs instead of using the classification outputs of the CNN as they observe higher performance with SVMs. In addition, they train a linear bounding box regressor to refine the coordinates of each region of interest, following the most recent advances with DPM. OverFeat [3] was another attempt to solve detection with CNN. Although it did win the ImageNet Detection Challenge in 2013, it is largely outperformed by R-CNN and the corresponding paper was never published. Following R-CNN, the first author Ross Girsick proposed two successive extensions. Firstly, Fast R-CNN [31], mainly improves over R-CNN in terms of speed. It introduces a Region of Interest (RoI) Pooling layer which extracts RoI features directly from the features maps of the entire image. This is largely inspired by Spatial Pyramid Pooling [32] which consists in pooling the features of an RoI with multiple binning resolutions and concatenating the outputs. RoI Pooling saves a lot of unnecessary forward passes through the CNN (R-CNN performs a forward pass for each RoI). Then, they dropped the SVM classifiers for the CNN outputs and integrated a bounding box regressor at the end of the CNN as a parallel branch to the classification layer. The training is done in a similar fashion as in R-CNN, they simply added a regression loss function, computed only on the positive proposals to train the bounding box regression branch. Secondly, Faster R-CNN [33] introduced the Region Proposal Network (RPN) to replace Selective Search as the proposal generation algorithm. Selective Search provides almost exhaustive proposals but is slow. The RPN is a lightweight CNN that densely predicts proposal coordinates and an objectness score at each position in the feature map. The box predictions are done as coordinate shifts from a set of pre-defined anchor boxes. After objectness filtering, this produces a reasonable number of proposals that can be processed with Fast R-CNN. The RPN is trained like Fast R-CNN with a similar loss function. A binary classification loss as the RPN classifies each location between objects or background (with the objectness score), and a regression loss. Example selection remains unchanged compared to Fast R-CNN. The training of both the RPN and Fast R-CNN is done in an end-to-end manner. Faster R-CNN achieves superior performance compared to its predecessors, but most importantly, it unlocks real-time detection with deep learning-based two-stage approaches.
## One-Stage Detectors

One-stage detectors appeared slightly later than two-stage ones. The reason for this is probably because two-stage models were the logical continuity of the sliding-window-based older detectors. These approaches are highly inefficient as they process parts of the input images many times. While this is reduced in modern two-stage detectors, they still have redundancies that limit their inference speed. In 2015, Redmon et al. proposed You Only Look Once (YOLO) [34], a first detector to avoid all redundancy as it only needs to look once at each part of the image. The main idea behind YOLO is to reformulate OD as a regression problem and not a classification one. Prior detectors solve OD by classifying regions of the input image, i.e., classifying given a region of interest. YOLO instead regresses the box coordinates and classifies the object jointly.

The main principle of YOLO is to split the input image into an $S \times S$ grid and predict bounding boxes, confidence scores and class probabilities for each cell in the grid. Each cell is "in charge" of detecting objects that are located within its boundaries. To deal with cases where more than one object is visible in one cell, $B$ bounding boxes and confidence scores are predicted per cell ($B = 2$ in the original paper). To keep model size constrained, the class probabilities are predicted only once per cell. This assumption limits the model to predict boxes of one class only per cell. The YOLO architecture is based on a deep CNN followed by two fully connected layers. The grid separation is directly implemented inside the architecture since the input size is fixed. YOLO is trained in an end-to-end fashion with a typical detection loss function. This loss function includes a regression part for box coordinates and a classification part, both implemented as L2 loss functions. Just like other object detectors, YOLO has an example selection strategy to compute its loss. Each ground truth box is attributed to the cell where its center is located and then to the box with the highest IoU. Thus, YOLO is extremely fast compared to the two-stage approaches (50 to 100 fps depending on the configuration for YOLO compared to less than 15 fps for Faster R-CNN). However, this speed improvement comes at the cost of slightly lower detection quality.

Just like the R-CNN family, YOLO was extended several times by its original authors and even later perpetuated by other research groups. In YOLOv2 [35], several improvements are introduced, including a lighter and fully convolutional architecture, decoupled class probabilities for each box, anchors boxes as in Faster R-CNN and a hierarchical word structure for refined classification. It also proposes several tricks and loss function adjustments to stabilize training. YOLOv2 hence achieves both higher detection performance and speed. YOLOv3 [36] is then introduced in an unpublished paper by the same authors, presenting several incremental improvements. Again, it outperforms its previous version both in quality and speed. After YOLOv3, its authors decided to quit the Object Detection research for ethical reasons, but many other groups continued to refine the YOLO framework. The race for the best performance and speed includes numerous versions of YOLO: YOLOv4 [37], PP-YOLO [38], YOLOX [39], YOLOv6 [40], YOLOR [41] and YOLOv7 [42]. Each of these works has its share of marginal changes involving elaborated loss design, structure change, augmentation
techniques and refined anchors generation. Note that YOLOv5¹ and YOLOv8² also exist but only as popular code repositories on GitHub, without any detailed report about their contributions.

YOLO models have a rich development history but are now reduced to marginal changes and implementation tricks. This is extremely useful from an engineering view, but it is less relevant from a research perspective. Nevertheless, the YOLO framework inspired plenty of other one-stage detectors. In particular, some detectors drop the use of anchors boxes and instead detect objects with keypoints. CornerNet [43] for instance produces heatmaps to determine the position of two corner points for each object, preventing the use of boxes at all. CenterNet [44] is a refinement of Corner-Net that only outputs a center point and infer the box dimensions from the keypoint features. Both CornerNet and CenterNet output a keypoint heatmap for each class and involve sophisticated post-processing to obtain the predicted bounding boxes. FCOS [45] simplifies this by directly predicting boxes from each feature location.

It is also worth mentioning Single Shot Detector (SSD) [46], which was proposed slightly after YOLO. It is also a one-stage detector, but unlike the first YOLO version, it is fully convolutional. This has several advantages as it predicts boxes densely on the images (higher recall, better detection of small objects) and it adapts better to different input image sizes. But most importantly, SSD leverages features from various scales for the predictions which dramatically improves the detection of small targets. Although this idea was introduced by SSD, it was popularized later with Feature Pyramid Networks (FPN) [1]. We will discuss these advancements in the following section as well as the choice of the CNN architecture choice.

One-stage object detectors were at first lagging behind two-stage detectors in terms of detection performance. However, the recent progress tends to close this gap, making the one-stage detectors the standard choice in the industry as they offer the best speed/performance tradeoff.

## Backbone network choice

In the OD literature, it is common to denote the main features' extractor CNN as the *backbone* of the network. Then, the lightweight module designed for classification and box regression on top of the backbone is logically called the *detection head*. What is placed between the backbone and the head (e.g., FPN and RPN) are sometimes referred to as the *neck*. Fig. 2.5 highlights these three main components of the object detector structure and outlines some design choices for each component. The backbone has an extremely important role in object detection as it extracts the features on which the classification and regression modules work. The choice of the backbone has been driven by the advances in classification, specifically the most common backbones have largely proven their capacities on ImageNet. First AlexNet [29] was used by R-CNN, then VGG networks [47] for Fast R-CNN, Faster R-CNN, YOLO, SSD and many others. These were quickly replaced by Residual Networks (ResNets) [48] which provide a large improvement in ImageNet classification, and consequently in

<footnote n="1">¹https://github.com/ultralytics/yolov5
²https://github.com/ultralytics/ultralytics</footnote>
the detection task. Following the extensions upon ResNets, object detectors successfully adopted WideResNet [49], ResNext [50] or EfficientNet [51]. More recently, the backbone network shifted from CNN to visual transformers, but this review will be conducted in Sec. 2.1.3.3.

Now, the backbones alone are not sufficient to extract relevant features for Object Detection. Backbone networks are originally designed to deal with curated images where one main object is visible and often located at the center of the image. Thus, backbones are not well-suited to deal with objects of various sizes and locations. An alternative to this issue is to leverage pyramidal features. This is not a very innovative idea as this was largely employed by face detectors during the 1990s and HOG models later (see Sec. 2.1.3.1). The actual innovation is to integrate this inside the CNN architecture. This was introduced first in SSD [46], but it was popularized with Feature Pyramid Networks (FPN) [1]. FPNs combine features with various resolutions from the input image at a single resolution (i.e., it is not necessary to perform the forward pass on multiple rescaled versions of the same image). FPNs extract intermediate feature maps in the backbone and aggregate them in a bottom-up manner (in contrast to the top-down processing of the forward pass). This bottom-up computation path is generally implemented with deconvolution layers, such as in Deconvolution SSD [52]. FPNs are plug-and-play modules that can be attached to most backbone networks and significantly improves the detection performance, especially for small objects. In two-stage detectors, FPN are often combined with Region of Interest Alignment Layer (RoI Align) to extract RoI features from the most relevant feature level (i.e., according to the RoI size). RoI Align was introduced in Mask R-CNN [53], an extension of Faster R-CNN for Instance Segmentation, that uses a FPN. In one-stage detectors, the detection is carried out on all feature levels output by the FPN, deeper levels are responsible for detecting larger objects.

Of course, plenty of contributions were proposed to improve upon FPNs. Path Aggregation Net (PANet) [54] for instance adds another top-down path before aggregating features from multiple levels with an Adaptive Feature Pooling layer. However, the design of the FPN architecture is not trivial and requires lots of trial and error. To find optimal FPN designs, NAS-FPN [55] proposed to apply principles of Neural Architecture Search for the design of FPNs and achieved superior detection performance. However, these questions are not so relevant to us as they mainly focus on Auto-ML problematics.

### Non-Maximal Suppression

Note that these methods often output numerous detections and require elaborated filtering schemes to prevent duplicates. Among others the Non-Maximal Suppression (NMS) became quite popular. In the case of largely overlapping boxes (i.e., when the IoU is above a fixed threshold), NMS keeps only the most confident box to prevent duplicates. This is done separately for each class so that objects from different classes may overlap. It improves the visual quality of the detection, but it can slightly degrade the detection performance when dealing with crowded scenes. For instance, Faster R-CNN employs the NMS both on the outputs of the RPN and on the final set of bounding boxes.
<figure><img src="image_44.png" /><figcaption>Figure 2.5: Possible architectural choices for Object Detector design.</figcaption></figure>

### 2.1.3.3 Recent Advances in Object Detection

A Transformer is a network architecture based on a multi-head self-attention mechanism. It was proposed in the context of Natural Language Processing (NLP) in 2017 by Vaswani et al. [56]. Since then, it became an essential component of most NLP applications. The original idea behind transformers is to represent the relations between different words in a sentence. For instance, the subject and a pronoun in a phrase must be strongly connected as they designate the same object. The self-attention mechanism from the Transformers is specifically built to adaptively compute these relations between words. As Transformers reformed the entire NLP field, vision models started to embed similar mechanisms to model long-range dependencies between parts of an image. Vision Transformers (ViT) [57] and Image Transformers [58] are one of the first attempts to solve image classification with Transformers and achieve considerable improvements over CNN baselines. They achieve this by dividing an input image into several patches that they treat just like words in a sentence.

Consequently, most vision tasks were quickly influenced by this new architecture. Object detection is no exception and in 2020, DETection with TRansformers (DETR) [59] is introduced as a first attempt to solve object detection using visual Transformers. Due to the time complexity of the transformer blocks ($O(H^2W^2)$, where H and W are the height and width of the image respectively), it is unreasonable to directly apply ViT for OD as input images are generally quite large. Instead, DETR leverages a ResNet backbone to extract relevant features but implements the detection head with transformers blocks. The head is divided into two parts, an encoder and a decoder. The encoder combines backbone features with positional encodings (i.e., fixed vector whose role is to keep track of the location of the patch processed). The decoder takes as input a set of object queries, learnable vectors that represent various positions in the images (similarly to positional encodings). Their role is to condition the detection toward a specific part of the image. Encoded image features are integrated inside the decoder through cross-attention layers. Finally, a light MLP predicts the box coordinates and class for each object query. The training of DETR is similar to prior object detectors (i.e., a classification and a regression loss function). However, the matching between predicted and ground truth boxes differs. DETR tackles OD as a set prediction problem, i.e., it predicts a set of bounding boxes as a whole and compares it with the set of ground truth boxes. This differs from
prior detectors which often employ box-to-box matching. In DETR, the matching is done by finding an optimal permutation of the sets (with the Hungarian algorithm) according to a cost involving boxes' positions and class labels.

Although DETR is a significant milestone in the detection landscape, it does not yield impressive performance gains over existing work. It even has considerable drawbacks, its inference is slow, it struggles with small objects and its training is one order of magnitude slower than prior detectors. Fortunately, several extensions mitigate these issues. First Deformable DETR [60], reduces the convergence time and improves detection performance with a deformable attention module. Deformable attention drastically reduces the amount of computation required and can process images with higher resolution. Deformable Attention Module is the twin of Deformable Convolutions [61] but for transformers. Similarly, H-Deformable-DETR [62] builds upon Deformable DETR with improved matching techniques which accelerate training further.

While the previous methods leverage transformers to make detections, they still rely on large CNN models as the backbone. This choice is also questioned by recent advances in visual transformer architectures. On the one hand, Data-efficient image Transformers DeiT [63] and Bidirectional Encoder representation from Image Transformers (BEiT) [64] both improve ViT's accuracy and training strategy. Both DeiT and BEiT show similar fine-tuning properties as CNNs, unlocking their application for various downstream tasks including object detection. On the other hand, ConViTs [65] and Swin Transformers [66, 67] make the attention computation much faster with spatial inductive bias and hierarchical structure respectively.

Another source of improvement for the detection backbones comes from self-supervised training. Recent advances in large-scale self-supervised training for classification are now being adapted to other visual tasks. DINO [68, 69] pre-trains both CNN and Transformer based backbones in a contrastive way with carefully designed augmentation schemes to obtain more robust visual features. Using these pre-trained backbones generally boosts the performance on many visual tasks, at least when applied to sufficiently similar images. The very recent Segment Anything Model (SAM) [70] also falls under the same category called *foundation models*. Even though no derivative work has been published yet, the capacities of SAM are promising for object detection.

Thanks to this progress, transformers-based backbones are about to replace CNN in most computer vision applications, including Object Detection. Nevertheless, some CNNs backbones are still proposed and seem to keep up with the rapid progress of transformers-based backbones. Among these works, ConvNeXt [71] brings several improvements over the original ResNet architecture to outperform Swin Transformer backbones. Closely related, InternImage [72] proposes an extension of Deformable Convolution to scale CNNs architectures as much as transformers (which was limited before). Even if the current hype is directed toward Transformers-based backbones, CNNs are not defeated yet. As an example, DynamicHeads [73] is a recent object detector based on a ResNext backbone achieving very competitive results on the COCO dataset. It leverages attention mechanisms
inside the detection head, but not Transformers modules as in DETR. Another recent detector based
on a CNN backbone is DiffusionDet [74]. It adapts the diffusion models (currently very popular for
image generation) to box prediction and obtains convincing performance as well.

To summarize, recent advances in Object Detection have largely followed the Transformer "revolu-
tion". First, with improved detection heads (the DETR family), and then, with improved backbones,
based on Transformers but also revamped CNNs.

2.1.3.4 Object Detection on Aerial Images

Most of the works presented in the above sections focus on the object detection task applied to
natural images. Aerial images differ significantly from natural images, they do not contain any
perspective, objects can be arbitrarily rotated, and they have a greater object size variance. Given
this, it seems obvious that some adjustments are required to adapt popular detectors to aerial images.

Oriented Bounding Boxes

As objects can be oriented in any direction, some aerial object detection datasets give annotations
as oriented bounding boxes. This slightly changes the problem, but most detectors can easily be
extended to deal with rotated boxes. The bounding box formulation can be extended so that it is
rotated, the regression layer must then be adapted to predict a rotation angle [75, 76], more than
four coordinates [77, 78], or to use rotated RoI, for instance with RoI Transform [79].

Small Object Detection

Aerial images contain objects with great size variance due to discrepancies in the shot conditions (altitude, sensor resolution, camera focal length, etc.). In addition, they also have smaller objects than natural images. To deal with the object size variance, it is possible to leverage supplementary information such as the ground sample distance (GSD). The ground sample distance represents the size of one pixel on the ground. Based on GSD, a model can infer the size of the RoI and therefore refine its predictions, as done by GSDet [80]. However, object size variance is generally a limited issue compared to detecting small objects, which remains an open challenge in object detection. Many attempts were made to solve this issue using specific architecture design [81], multiscale training [82, 83], data-augmentation [84] or super-resolution [85, 86, 87]. Additionally, Normalized Wasserstein Distance (NWD) [88] proposes an alternative to the IoU loss specifically designed for detecting small objects. It consists in computing the Wasserstein distance between two Gaussian distributions fitted on the two bounding boxes compared. Moreover, NWD is not only used as a loss function but also as an example selection criterion.

2.1.3.5 Training Object Detectors

While we briefly presented how object detectors are trained in the previous sections, we did not give much detail about the loss functions and the optimization process. We remedy this here by reviewing the loss functions of several popular object detection benchmarks.
Loss functions for Object Detection

As object detectors must solve both classification and regression tasks, most detection loss functions are divided into two components, a classification loss and a regression loss. Plenty of choices exist for both components. For the classification loss, the most common choice is the Cross-Entropy loss:

$$
\mathcal{L}_{\text{CE}}^{\text{cls}}(\hat{\mathbf{y}}_i, \mathbf{y}_i) = -\log(\hat{l}_i^{c_i}), \qquad (2.13)
$$

where $\hat{l}_i^{c_i}$ is the predicted probability that the box $i$ contains an object of class $c_i$, $c_i$ being the true label ($\mathbf{y}_i = (b_i, c_i)$). However, some alternatives such as the L1 or L2 losses over the class probabilities are also employed:

$$
\mathcal{L}_{\text{L1}}^{\text{cls}}(\hat{\mathbf{y}}_i, \mathbf{y}_i) = \|\hat{l}_i - l_i\|_2^2, \qquad (2.14)
$$

where $\hat{l}_i$ denotes the class probability vector ($\hat{l}_i = \{\hat{l}_i^c\}_{c \in C}$) and $l_i$ is the one-hot encoded true probability vector of box $i$. Another very common classification loss function in recent detector is the Focal Loss (FL) function [89], which was introduced with the RetinaNet object detector. Focal loss is designed to address the class imbalance issue that is inherent to dense object detectors (the background class is much more represented than other classes). Focal loss reduces the relative loss of well-classified examples so that the learning process focuses on misclassified objects. It is defined as follows:

$$
\mathcal{L}_{\text{FL}}^{\text{cls}}(\hat{\mathbf{y}}_i, \mathbf{y}_i) = -\alpha_{c_i} (1 - \hat{l}_i^{c_i})^{\gamma} \log(\hat{l}_i^{c_i}), \quad (2.15)
$$

where $\alpha_{c_i}$ is an inverse class-frequency parameter and $\gamma$ controls how much FL reduces the contribu-
tion of well-classified examples to the loss. In RetinaNet, the authors leverage only a binary version
of FL as they tackle the multi-class classification problem as $|C|$ binary classification problems. They
replaced the Softmax activation function on the classification layer with a Sigmoid activation and
classified the box as either background or foreground for each class independently. This binary
formulation of the classification task will be extensively re-used by derivative detectors.

For the regression part, a greater variety of loss functions exists in the literature. L1 and L2 losses
are common in early CNN-based detectors. The Smooth L1 (or Huber Loss [90]) is a variant of the
L1 loss leveraged by the Faster R-CNN family of detectors. It combines the L1 and L2 norms to get
a smooth loss function around 0. Next, UnitBox [91] introduced the IoU loss function as the new
standard for box regression training:

$$
\mathcal{L}_{\text{IoU}}^{\text{reg}}(\hat{\mathbf{y}}_i, \mathbf{y}_i) = - \log(\text{IoU}(\hat{b}_i, b_i)), \qquad \text{Log version} \qquad (2.16)
$$

$$
\mathcal{L}_{\text{IoU}}^{\text{reg}}(\hat{\mathbf{y}}_i, \mathbf{y}_i) = 1 - \text{IoU}(\hat{b}_i, b_i). \qquad \text{Linear version} \tag{2.17}
$$

Following the IoU loss, several extensions were proposed, e.g., Generalized IoU [92], Distance-IoU
[93], or α-IoU [94], these will be reviewed later in Chap. 8. To summarize, Tab. 2.1 gives an overview
of the loss functions used by common object detectors.
<table><caption>Table 2.1: Summary of the loss functions used in several object detection frameworks.</caption><thead><tr><th></th><th>Classification</th><th>Regression</th></tr></thead><tbody><tr><td>Faster R-CNN [33]</td><td>Cross-Entropy for the detection head<br>Binary Cross Entropy for the RPN</td><td>SmoothL1 Loss for head and RPN</td></tr><tr><td>YOLO [34]</td><td>L2 Loss on class probability vector (for grid cell containing an object)<br>L2 Loss on true class probability (for all cells)</td><td>L2 Loss on box center<br>L2 Loss on square root of box dimensions</td></tr><tr><td>RetinaNet [89]</td><td>Binary Focal Loss</td><td>SmoothL1 Loss</td></tr><tr><td>UnitBox [91]</td><td>Binary Cross Entropy</td><td>IoU Loss (log)</td></tr><tr><td>FCOS [45]</td><td>Binary Focal Loss</td><td>GIoU Loss (linear)</td></tr><tr><td>DETR [59]</td><td>Cross Entropy</td><td>L1 Loss and GIoU Loss</td></tr><tr><td>DiffusionDet [74]</td><td>Binary Focal Loss</td><td>L1 Loss and GIoU Loss</td></tr></tbody></table>

## Example selection strategy

In the previous paragraph, we presented the various loss functions employed in the Object Detection literature. For simplicity, we defined these losses for a couple of predicted and ground truth detection labels ($\hat{y}_i, y_i$). In reality, the losses are computed as the sum of all such couples (over one or multiple images). However, it is not straightforward to build these couples as there may be more predictions than ground truths, missed objects, or false detection. Each detector has its own strategy to operate the matching between prediction and ground truth. These strategies were briefly presented in the previous sections, but we regrouped them inside Tab. 2.2 for clarity.

<table><caption>Table 2.2: Brief description of some existing prediction ground truth matching strategy in existing object detectors.</caption><thead><tr><th></th><th>Matching Strategy</th></tr></thead><tbody><tr><td>Faster R-CNN [33]</td><td><ul><li>Select RoI with at least 0.5 IoU with a GT as Positive Samples (PS) and RoI with low IoU (&lt; 0.1) as Negative Samples (NS).</li><li>Classification loss is computed on all selected samples (PS with the corresponding GT class and NS with the background class).</li><li>Regression Loss is only computed with PS.</li></ul></td></tr><tr><td>YOLO [34]</td><td><ul><li>Select PS as grid cells in which there is a GT center point and assign the highest IoU boxes in case of multiple GT in one cell.</li><li>All others are NS.</li><li>Classification done separately on PS and NS.</li><li>Regression loss with PS only.</li></ul></td></tr><tr><td>FCOS [45]</td><td><ul><li>Select PS as feature map locations that fall inside a GT, all others are NS.</li><li>If multiple GT for the same location take the smallest one.</li><li>Classification on PS and NS.</li><li>Regression on PS only.</li></ul></td></tr><tr><td>DETR [59]</td><td><ul><li>1-to-1 optimal prediction and GT assignment according to localization and classification cost.</li><li>No-object are added to the GT set when the predictions are more numerous.</li><li>Classification loss is computed for all matched couples.</li><li>Regression loss only for couples with an actual GT.</li></ul></td></tr><tr><td>H-DETR [62]</td><td><ul><li>DETR matching.</li><li>Supplementary 1-to-many matching with duplicated and augmented GT for training.</li></ul></td></tr><tr><td>DiffusionDet [74]</td><td><ul><li>DETR matching</li></ul></td></tr></tbody></table>
### 2.1.4 Datasets Presentation

There exist numerous object detection datasets in the literature. We present in this section four of them in detail as they will be extensively used in this manuscript. These datasets are Pascal VOC [5], MS COCO [6], DOTA [77] and DIOR [95]. We choose these datasets because they were the most relevant and widespread datasets of natural (Pascal VOC and COCO) and aerial (DOTA and DIOR) images at the beginning of this project. Some other datasets will be punctually used, especially for the cross-domain experiments in Chap. 7, they will be presented in detail there. However, we draw up a non-exhaustive list in Tab. 2.3 of the most well-known object detection datasets in the literature.

<table><caption>Table 2.3: Overview of existing detection datasets</caption><thead><tr><th>Image Type / Application</th><th>Dataset Name</th><th># Classes</th><th># Images</th><th># Instances</th></tr></thead><tbody><tr><td rowspan="4">Natural</td><td>Pascal VOC [5]</td><td>20</td><td>11.5k</td><td>27k</td></tr><tr><td>COCO [6]</td><td>80</td><td>117k</td><td>1.5M</td></tr><tr><td>LVIS [96]</td><td>1203</td><td>100k</td><td>1.3M</td></tr><tr><td>Object365 [97]</td><td>365</td><td>2M</td><td>30M</td></tr><tr><td rowspan="2">Autonomous Vehicle</td><td>KITTI [98]</td><td>11</td><td>7k</td><td>80k</td></tr><tr><td>BDD100k [99]</td><td>10</td><td>400k</td><td>3M</td></tr><tr><td rowspan="3">Pedestrian</td><td>CityPerson [100]</td><td>1</td><td>3k</td><td>19k</td></tr><tr><td>TinyPerson [101]</td><td>1</td><td>1610</td><td>72k</td></tr><tr><td>CrowdHuman [102]</td><td>1</td><td>15k</td><td>340k</td></tr><tr><td rowspan="5">Aerial</td><td>COWC [103]</td><td>1</td><td></td><td>33k</td></tr><tr><td>DOTA [77]</td><td>16</td><td>2.8k (megapixels)</td><td>220k</td></tr><tr><td>DIOR [95]</td><td>20</td><td>23k</td><td>190k</td></tr><tr><td>xView [104]</td><td>60</td><td>1.1k (megapixels)</td><td>1M</td></tr><tr><td>FAIR -1M [105]</td><td>37</td><td>15k (megapixels)</td><td>1M</td></tr><tr><td rowspan="2">Agricultural / Food</td><td>DeepFruits</td><td>7</td><td>457</td><td>2.5k</td></tr><tr><td>Oktobeerfest [106]</td><td>15</td><td>1k</td><td>2.5k</td></tr><tr><td rowspan="4">Other Modalities</td><td>ClipArt [107]</td><td>32</td><td>5k</td><td>13k</td></tr><tr><td>LogoDet [108]</td><td>3000</td><td>159k</td><td>194k</td></tr><tr><td>SIXray [109]</td><td>6</td><td>9k</td><td>1M</td></tr><tr><td>DroneVehicle [110]</td><td>5</td><td>56k</td><td>191k</td></tr></tbody></table>

#### 2.1.4.1 Natural Images

Natural images are the kind of image humans are the most familiar with, therefore it is logically the most common application in Computer Vision. Object detection is no exception and most proposed detectors are developed to process natural images. Hence, our analysis must be conducted as well on natural images even though the industrial interest of COSE is more towards aerial imagery. To this end, we choose Pascal VOC and MS COCO as our main sources of natural images.

**Pascal VOC [5]** – The Pascal VOC challenge took place every year between 2005 and 2012. This competition defined the object detection problem as we know it today. The last version of the dataset
Pascal VOC 2012 includes images of various sizes and aspect ratios. Each image is annotated with horizontal bounding boxes around objects belonging to 20 classes. Examples of images and a list of classes, ordered by the number of occurrences, are available in Fig. 2.6a.

**MS COCO [6]** – MS COCO is an extension of Pascal VOC which includes much more images and classes. The set of images is completely distinct from Pascal VOC, but all classes in Pascal VOC are included in COCO. Similarly, Fig. 2.6b presents image examples and a list of COCO classes.

#### 2.1.4.2 Aerial Images

The overall goal of this project is to detect objects from aerial images. Aerial images are sometimes associated with low-altitude drone images. These images are halfway between natural and aerial images as they often preserve some perspective. Remote Sensing Images (RSI), i.e., acquired from planes or satellites with nadir-oriented cameras are much closer to COSE's application. In this manuscript, we refer to this kind of image both as aerial or RSI images. There exist a few publicly available datasets of such images. We have chosen two of them based on the ground resolution of the images (in agreement with COSE systems) and their availability at the beginning of this project.

**DOTA [77]** – DOTA contains images coming from Google Earth and distinct satellites Jilin-1 and Gaofen-2 (with roughly 1m spatial resolution GSD). Images from DOTA are large, ranging from 800 to 4000 pixels in width and objects are annotated with oriented bounding boxes. To ease the handling of the images, we prepared DOTA by tiling all images into 512 × 512 patches with a 50% overlap and converted the annotations to horizontal bounding boxes. Fig. 2.7a presents images and the class list for DOTA.

**DIOR [95]** – DIOR is very similar to DOTA. It contains only images scrapped from Google Earth and has slightly more classes than DOTA. The images are already tiled at 800 × 800 pixels and boxes are horizontal. Fig. 2.7b presents images from DIOR and the list of classes.

## 2.2 Few-Shot Learning: Learning with Limited Data

As presented in the introduction, COSE faces a substantial challenge in the design of its imaging systems: the lack of real-case images and unknown objects of interest. All methods described in Sec. 2.1 require large annotated training sets to achieve reasonable detection performance, which is misaligned with COSE's constraints. This issue is common in the industry, most computer vision problems lack large annotated datasets, and therefore the direct application of research contributions is often challenging. Fortunately, there exists an entire research field dedicated to learning with limited annotated data. The main paradigm in this field is to learn a closely related task with sufficient data and adapt to the real task with limited annotations. Two kinds of adaptation can be considered: class adaptation and domain adaptation. Given a computer vision task such as classification, the former consists in learning to classify objects or images among a set of classes and then adapt to another set of classes. This is usually called Few-Shot Classification (FSC). While classifica-
tion is not the primary interest of COSE, it is worth exploring the FSC literature as it is an older field, much more developed than FSOD and because FSC lays the foundation for tackling more complex tasks in the few-shot regime. On the other hand, domain adaptation consists in adapting to different kinds of images, e.g., different seasons, weather conditions, general environments, etc. In the strict definition of domain adaptation, the classes of interest remain the same. However, the setting when both the classes and the domain change is also studied in the literature. It is more challenging, but it better reflects the industrial needs such as COSE's. In this section, we review both kinds of adaptations for the classification problem. Even though it is not a task of interest for COSE, understanding few-shot adaptation strategies is crucial before addressing the more challenging problem of Few-Shot Object Detection which we reserve for Chap. 3.

## 2.2.1 Few-shot Classification

### 2.2.1.1 Problem Definition

Classification is a simpler problem than detection. Given a set of classes $C$ and an input image $I$, one wants to find the class $c \in C$ that is depicted by $I$. Of course, the higher considerations briefly presented in Sec. 2.1.1 about how to properly define the membership of an image to a class still holds. For classification as well, the class membership is determined by human appraisal and common sense. Solving a classification task is to find a model $\mathcal{F}(\cdot, \theta)$ that outputs a class label for a given input image $I$:

$$ \mathcal{F}(I, \theta) = \hat{c} \in C. \qquad (2.18) $$

Deep Learning based models proved to be particularly adapted to the classification task in a fully supervised setting (i.e., provided with sufficiently large annotated datasets). This was supported amongst others by LeNet [111] for digit classification, and by AlexNet [29] and ResNet [48] for ImageNet classification. However, the classification task in this form is not a topic for this section, and we refer to [112] for a complete review of existing works in this field.

In the few-shot setting, the classification goal remains the same, predicting the class of an image. The input image to an FSL model is usually denoted as a *query image*, and therefore, the test set is called the *query set*. What changes between the few-shot and regular settings is the amount of annotated images available to train the model. In the literature, the expression *N*-way *K*-shot classification designates the task of classifying images amongst *N* different classes only provided with *K* annotated examples per class. The *NK* images constitute the *novel dataset*, in contrast to the *base dataset* which contains an arbitrary number of annotations for another set of classes. In the few-shot literature, the *novel dataset* is often called the *support set*, and its elements *support examples*. Similarly, the sets of classes of the *base* and *novel* datasets are called the *base classes set*.
<figure><img src="image_45.png" /><figcaption>(a) Examples of Pascal VOC images and class repartition on the training split.</figcaption></figure>

<figure><img src="image_46.png" /><figcaption>(b) Examples of MS COCO images and class repartition on the training split.</figcaption></figure>

<caption>Figure 2.6: Image examples for the Natural images dataset Pascal VOC and MS COCO.</caption>
<figure><img src="image_47.png" /><figcaption>(a) Examples of DOTA images and class repartition on the training split.</figcaption></figure>

<figure><img src="image_48.png" /><figcaption>(b) Examples of DIOR images and class repartition on the training split.</figcaption></figure>

<caption>Figure 2.7: Image examples for the Aerial images dataset DOTA and DIOR.</caption>
(noted $C_{\text{base}}$) and novel classes set (noted $C_{\text{novel}}$) respectively. Specifically, we have:

$$ \mathcal{D}_{\text{base}} = \{(I_i, c_i)\}_{1 \le i \le |\mathcal{D}_{\text{base}}|} \quad c_i \in \mathcal{C}_{\text{base}}, \qquad (2.19) $$

$$ \mathcal{D}_{\text{novel}} = \bigcup_{c \in \mathcal{C}_{\text{novel}}} \{(I_k^c, c)\}_{1 \le k \le K}. \qquad (2.20) $$

As mentioned above, $\mathcal{D}_{\text{base}}$ is used to train the model during a first phase called *base training*. During this phase, the model has access to plenty of annotated data and is trained in a supervised manner to classify images within $C_{\text{base}}$. It is noteworthy to point out that this supervised base training is not the only possible choice. Recent advances in Self-Supervised Learning (SSL) [113, 114, 115, 116] proved that SSL is a competitive alternative to supervised base training.

After base training, the novel dataset is leveraged to adapt the model to classify the novel classes. Hence, the few-shot classification task can be seen as predicting the class label from the input image and the novel dataset:

$$ \mathcal{F}(I, \mathcal{D}_{\text{novel}}) = \hat{c} \in \mathcal{C}. \qquad (2.21) $$

The model adaptation generally starts with small architectural modifications, such as replacing the final classification layer with a novel layer randomly initialized and with the right number of outputs (e.g., if the numbers of base and novel classes differ). Then, several approaches exist for adjusting the model to the novel classes given the novel dataset. We identify here four different adaptation strategies and will present each of them in the next sections. These strategies are: fine-tuning, metric-learning, meta-learning and attention-mechanisms. However, there are no clear boundaries between these four areas, Fig. 2.8 illustrates the interactions between the various strategies and gives a few examples for each category. We propose this taxonomy as it suits well the few-shot object detection field. Hence, reviewing FSC through this lens helps to understand how these techniques could be extended for detection. However, there exist much more detailed taxonomies and reviews about FSC in the literature, [117, 118] are worthy examples. Note that the novel dataset can also be used during inference, so that adaptation is done "on the fly". This is called *transductive inference* and will be presented in Sec. 2.2.1.6.

In the most common few-shot setting, we have $C_{\text{base}} \cap C_{\text{novel}} = \emptyset$, meaning that there are no common classes between the base and novel sets. Of course, some works focus on relaxing these assumptions, we will outline some of them in Sec. 2.2.1.7

### 2.2.1.2 Fine-tuning

Probably the most straightforward way to tackle the FSC task is to employ fine-tuning or transfer learning. This method trains the model on $\mathcal{D}_{\text{base}}$ and then the model weights are fine-tuned using $\mathcal{D}_{\text{novel}}$ with only the few examples available. It works well but the fine-tuned models are prone to strong overfitting and catastrophic forgetting [119]. Overfitting on the novel set is problematic as it means that the fine-tuned model will have poor generalization capabilities, i.e., its performance
<figure><img src="image_49.png" /><figcaption>Figure 2.8: Taxonomy of the Few-Shot Classification literature. To illustrate each category of the taxonomy, a few papers are selected as representatives among others. Papers marked with a * are not solving the FSC task but are included in this figure as no contribution in the literature tackles classification from this perspective.</figcaption></figure>

will be way lower on the test set than on the training set. Catastrophic forgetting is a more subtle issue. It happens when the performance of the fine-tuned model on the base classes drops. In the case of the simple FSC it is a subtle issue, but it becomes more challenging when dealing with extended setups such as generalized FSL and Continual Learning (see Sec. 2.2.1.7 for more details). However, the authors of [119] propose regularization penalties to be applied during fine-tuning that alleviate both the overfitting and catastrophic forgetting. Specifically, the penalty prevents the fine-tuned weights from being too far from the pre-trained weights. Similarly, [120] proposes several regularization loss functions and a grouped parameter update during fine-tuning to overcome the overfitting. Closely related, [121] leverages Direct Loss Minimization's theorem [122] to optimize the model's weights with an Information Retrieval Loss at inference. Although fine-tuning is a relatively simple approach for FSL, it was not much explored in the case of classification. We will see in Chap. 3 that it has gained more attention recently for more complex tasks.

### 2.2.1.3 Metric Learning

Metric learning is a branch of deep learning which consists of learning self-organized representation spaces, i.e., similar inputs should have similar representation in the embedding space. It was first introduced with Siamese Networks for signature verification [123] and later for face identification [124]. The idea behind the Siamese Networks is to leverage two copies of the same model and feed them two different images. The output of the two networks should be similar if the input
images look similar as well. Siamese networks were then applied for one-shot image classification by [125]. It was one of the first attempts to solve this task using deep neural networks. Features from the query and support images are extracted by the siamese nets and then compared by a final prediction layer. This final layer takes as input the difference between the features of the query and support images. Its role is to assess whether the features are similar enough so that the two images belong to the same class. Following Siamese Networks, a series of works was proposed based on the same principle. These contributions are embodied by Prototypical Networks [126]. ProtoNets replace the final prediction layer of the siamese networks with a linear classifier and extend the metric learning framework for multi-class and increased number of shots. Specifically, the features of all support examples of one class are aggregated to form *class prototypes* and query features are classified according to the class of the closest prototype. Many variants of the ProtoNets were then proposed. Inspired by Siamese Networks, Relation Networks [127] replace the linear classifier of the Prototypical Network with a small MLP trained to predict a similarity score based on the query features and a prototype. The difference with Siamese Networks is that this is done with each class prototype allowing Relation Nets to address multi-class problems. Other extensions include prototypes rectification for intra and extra-class variance [128], semi-supervised prototypes refinement [129] and multiple prototypes per class [130]. It is essential to note that Prototypical Networks and their extensions leverage episodic training strategies borrowed to meta-learning. This strategy consists in dividing the training into shorter *episodes*. During each episode, the model is trained for a random task, generally a subset of the novel classes (only the $D_{novel}$ dataset is considered by these approaches). The episodic strategy follows the "learning to learn" paradigm and mimics the adaptation process the model will undergo at test time.

The episodic strategy in the context of metric learning is first used in Matching Networks [131], an earlier work than Prototypical Network. Matching Networks are more inspired by the meta-learning techniques, hence the episodic strategy. Two networks are trained jointly, one to extract the support features and one for the query features. However, the way query and support features are combined differs from other metric-learning methods. The authors leverage an attention mechanism to compute the predicted class probabilities as a similarity-aware weighting of the support examples labels. Closely related, Task Dependent Adaptive Metric (TADAM) [132] learns a task representation and adapts its embedding network through a task-conditioning layer which resembles an attention mechanism. The class prototypes are then task-dependent and an image is classified according to the most similar prototype. Matching Network and TADAM are therefore at the intersection of three approaches for FSL: metric-learning, meta-learning and attention mechanism. These are reviewed in the two following sections.

### 2.2.1.4 Meta-Learning

As hinted at the end of the previous section, meta-learning's paradigm is "learning to learn". This was the main motivation for the episodic training strategy described there. Each episode forces the model to adapt to new classes, repeating these episodes should overall increase the adaptation
capacities of the model. However, the concept of meta-learning goes even further.

This concept was introduced and popularized during the 1990s [133, 134]. At the time, meta-learning was mostly applied in the context of policy learning, with evolutionary or reinforcement learning methods. It was brought up-to-date for the few-shot classification by Model-Agnostic Meta-Learning (MAML) [135] which proposes to directly train the initial weights of a classifier so that it will quickly adapt to a given task. The optimization is done in a nested manner. At the inner level, a task is sampled (like with the episodic training strategy) and the classifier is initialized with the current initial weights. A few gradient steps are performed on the classifier with respect to the task objective function. Then, at the outer level, the initial weights are updated through gradient descent on the task loss value computed with the trained classifier. The meta-update converges to a set of initial weights that make the classifier "easy to train" on any task. However, MAML does not take task information into account for the weight initialization, and it is unrealistic to find truly task-agnostic initializations. Therefore, [136] extends MAML to only choose a subset of the optimal initial parameters to initialize the classifier based on task information. Orthogonally, some contributions integrate uncertainty in the weight initialization [137, 138], others make the training easier [139, 140] or provide a theoretical framework and guarantees [141].

Similarly, some meta-learning techniques propose learning the optimization process instead of the weight initialization. This is the case of Optimization as a Model [142] which trains a LSTM meta-learner to output gradient updates for the classifier network. The meta-learner takes as input the weights of the classifier and the gradients computed on a given task. The recurrent nature of the meta-learner helps to keep track of the previous error signals and update consequently the weights of the classifier. Close to MAML, the meta-learner is updated after several weight updates (with different tasks) based on the loss values of the classifier on a test set.

Another meta-learning direction is introduced with Ridge Regression Differentiable Discriminator (R2D2) [143]. It consists in teaching a model "to use" standard machine learning tools such as Ridge or Lasso Regression. These techniques often have closed-form solutions and are fast to compute when few data are available. In R2D2, a CNN is trained as a feature extractor as a meta-learner, while the classifier's weights are computed with a ridge regression from the support set. The meta-learner CNN is trained to extract features that will generate optimal classifier weights through ridge regression.

### 2.2.1.5 Attention-Based methods

As an alternative to the rather complex and heavy meta-learning methods, a line of work followed the MAML principle but focused only on some layers rather than on the complete classifier. This originates with LearNets [144] that are trained to output the weights of a convolution kernel from a support example. The kernel is then used in a *Dynamic Convolution Layer* (DCL) inside the classifier, which in the end predicts a class-membership score (according to the class of the support example). When multiple classes are available in the support set, the DCL is applied with each class features
independently and the classifications are done in a binary fashion. This can be understood as an attention mechanism between the query and support features. The dynamic convolution acts as a filter responsive to the support features. To put it another way, the locations in the query feature map that are similar to the support features will be highlighted. The dynamic convolution sets the focus of the classifier on features from the support class. Self-Attention (SA) is probably the most common form of attention mechanism used in the current deep learning literature. It highlights similarity links between subparts of the input (e.g., different locations in an image, or two words in a sentence). Here, with DCL, this is done with two distinct images: a query and a support image. Several other works build upon this idea. Simple Neural Attentive Learner (SNAIL) [145] designs more complex attention blocks, based on transformers, to perform the classifier's adaptation. Although their primary goal is to tackle few-shot reinforcement learning with temporal convolutional layers (to deal with causality), they apply it successfully to FSC as well. The *NK* images in the support set are fed in random order and the query image is given last. Very similar to SNAIL, CrossTransformers [146] assemble an attention module to combine query and support features. The crucial difference with SNAIL is the preservation of the spatiality of the features. Most previous works aggregate the support features to perform the adaptation, losing the spatial information of the support image. Inspired by the recent progress of ViT, CrossTransformers manage to adapt the classifier while preserving spatial information. Just like many metric learning methods presented in the previous section, some of the attention-based techniques discussed above borrow an episodic training strategy from meta-learning.

The query-support attention mechanism can also be interpreted as a conditioning of the classifier's input based on the support features. That is the view adopted by Dynamic Conditional Network [147]. The general idea is very similar to LearNet except for the training which is not done in an episodic manner. Conditionally Shifted Neurons (CSN) [148] see the adaptation as the conditioning of the classifier activations. The meta-learner outputs shift values that are added to the pre- or post-activation values in the network. The shifts are computed from a *task description* stored in a memory. The task description regroups the activation of all the layers of the classifier fed with the support images. The use of a memory bank is widespread along with attention-based mechanisms for FSC. Memory Augmented Neural Network (MANN) [149] epitomizes this line of work. It leverages a *controller* (i.e., a small network) to read and write in the memory. The controller generates a key from an input, which is then used to either add a new entry in the memory or retrieve already stored information. The retrieval is done through an attention-like mechanism. The memory is built throughout a task episode adaptively. When a new memory is added, if a similar memory is already stored, the new memory refines the existing one to build more relevant representations. Plenty of contributions took inspiration from MANN. Adaptive Posterior Learning (APL) [150] refines the memory writing process to store only "surprising" memories. Meta Networks [151] also leverage external memory only accessible to the meta-learner in charge of adapting the classifier. [152] proposes a second *abstract memory* which stores refined information relevant for the current task. All these memories are generally wiped when the task is modified. However, *life-long memory*
[153] also benefits the FSC even though it is mostly exploited for slightly more challenging tasks such as Generalized FSL or Continual Learning (see Sec. 2.2.1.7).

### 2.2.1.6 Transductive Inference

A recent line of work tackles the few-shot classification problem with transductive inference. Transductive inference, in contrast to inductive inference, consists of leveraging labeled and unlabeled images and classifying all unlabeled data points at the same time. Conversely, inductive inference deals with each data point independently. Of course, deploying such methods in an industrial scenario requires having multiple data points available at test time. For real-time applications, it is generally not practical. However, in the case of COSE and the detection task, this is largely applicable. Very large images cannot be processed as a whole, they must be divided into smaller images. In addition, for the detection task, an image is often seen as a collection of objects or regions of interest. COSE's use case is therefore rather well-suited for transductive inference. Therefore, we briefly review the recent advances of transductive learning for FSC in this section.

Transductive inference is an old concept of statistical learning that was popularized under this name and for machine learning applications by Vladimir Vapnik in the 1990s [154]. As mentioned above, in the transductive setting, an entire unlabeled dataset (e.g., a test dataset or a query set in the few-shot context) is available at test time. Transductive methods leverage information contained both in the support and query set to make predictions. Before application on FSC, transduction was applied to regular classification on small-size datasets, yielding significant improvements over inductive methods. Amongst them, Transductive Support Vector Machines [155] extends the well-known SVM [156] to make use of unlabeled information to refine the class separation margins. Another direction is taken by [157] which derives an iterative method that propagates known labels to unlabeled data points according to their similarity. Recently, TransBoost [158] even applied transduction to the entire ImageNet dataset with significant accuracy gain over inductive methods. The authors propose a fine-tuning approach to refine trained neural networks to perform better on a specific test set. It takes both the training set and the unlabeled test set to compute a regularization loss function that penalizes similar images to be classified differently by the network.

Transductive inference is especially effective in the few-shot context as the limited labeled data is often not enough to provide sufficient supervision. Leveraging additional unlabeled data is therefore highly beneficial. Various approaches were proposed to make use of this supplementary information within the already existing few-shot frameworks. Probably the most straightforward approach is to fine-tune pre-trained models with additional regularization loss based on the labeled and unlabeled data. This is the direction taken by [159] which compares the few-shot performance of several methods against a simple transductive fine-tuned baseline. Similarly, Transductive Episodic-wise Adaptive Metric (TEAM) [160] and Transductive Information Maximization with Gradient Descent (TIM-GD) [161] also both leverage fine-tuning objectives to refine the model before transductive inference. This resembles semi-supervised learning which fine-tunes models with additional unla-
beled data, i.e., different from the training and test set. However, the essence of transductive learning lies more in adapting the inference based on the additional information rather than fine-tuning the model. This can be done by direct optimization of an objective function with regularization as in LaplacianShot [162], TIM [161] or Cross-Attention Networks [163]. Many propositions iteratively propagate the known labels to unlabeled data points within a graph structure [164, 165, 166, 167, 168]. But there also exist contributions that exploit transductive inference through metric learning with e.g., Prototype Rectification[128] or Meta-Confidence Transduction [169], which meta-learn a distance metric. Meta-learning based methods also get their transductive extension, such as Reptile [170] which extends MAML to perform transductive inference by leveraging information shared by test samples through the batch normalization layers.

### 2.2.1.7 Extending the Few-Shot Setting

The few-shot setup that we described in previous sections is limited and makes a few assumptions:

1. The set of base and novel classes are known in advance.

2. At test time, only the performance on novel classes matters.

3. Novel classes are only added once and all at the same time.

These assumptions significantly simplify the problem, but these are relaxed by different sub-fields of few-shot learning. In some aspects, the few-shot detection can be seen as a relaxation of these assumptions. Various tasks, similar to few-shot classification, exist in the literature. [171] provides a comprehensive taxonomy of these tasks. We will briefly present in this section some relevant extensions of the few-shot classification for COSE's application and the detection task. Tab. 2.4 provides an overview of these tasks and their differences in terms of goal and available data.

#### Few-shot Open-set Recognition

Open-set classification assumes that some classes are unknown during training (i.e., the training dataset is incomplete) and deals with these classes. Instances of unknown classes can be rejected or identified as unknown classes. It models real use cases better as test data can be contaminated by classes not included in the training set. Object detection can be assimilated as an open-set problem as objects belonging to a fixed set of classes must be localized while rejecting everything else as background. The training set can only contain a limited variety of background examples and new instances of background will be presented to the model at test time. There are plenty of approaches for Open-set Recognition, but we will not review them here in detail and refer the reader to a complete survey [171] about it. Instead, we simply outline the general principle behind algorithms that tackle this problem. Two main approaches coexist in the literature, discriminative and generative approaches. The former ones propose techniques to distinguish between known and unknown classes using discriminative information, e.g., distance to class representations [172]. The latter leverage generating models to hallucinate negative examples as additional training data [173]. Of course, transduction also helps in this case and *outlierness* score [174] can be computed using the unlabelled examples available at test time.
<table><caption>Table 2.4: Summary of the various flavors of classification tasks existing in the literature. The second column, classes of interest, denotes what is the overall goal of the task. The last column presents the possible class setup encountered both in the query and support set ($C_{query}$ and $C_{support}$ respectively.). $C_{unknown}$ represents additional classes that should be identified in the open-set setting. In the detection task, $C_{background}$ denotes all object classes that can be present in the background and that should not be detected.</caption><thead><tr><th>Task</th><th>Classes of interest</th><th>Novel supervision</th><th>Query-support interaction</th></tr></thead><tbody><tr><td>Regular Classification</td><td>C<sub>base</sub></td><td>None</td><td>None</td></tr><tr><td>Few-shot Classification</td><td>C<sub>novel</sub></td><td>K examples per novel class</td><td>C<sub>query</sub> = C<sub>support</sub></td></tr><tr><td>Zero-shot Classification</td><td>C<sub>novel</sub></td><td>External information (e.g., class labels)</td><td>None</td></tr><tr><td>Generalized FSC</td><td>C<sub>base</sub> &cup; C<sub>novel</sub></td><td>K examples per novel class</td><td>C<sub>query</sub> = C<sub>support</sub></td></tr><tr><td>FS Open-set Recognition</td><td>C<sub>novel</sub> &cup; C<sub>unknown</sub></td><td>K examples per novel class<br>None for unknown classes</td><td>C<sub>query</sub> &sub; C<sub>support</sub> or C<sub>query</sub> &supset; C<sub>support</sub></td></tr><tr><td>Continual Learning</td><td>C<sub>base</sub> &cup; (&cup;<sub>i</sub> C<sub>novel</sub><sup>i</sup>)</td><td>K examples per novel class</td><td>None</td></tr><tr><td>Few-Shot Object Detection</td><td>C<sub>novel</sub> \ C<sub>background</sub></td><td>K annotated images per novel class</td><td>C<sub>query</sub> = C<sub>support</sub></td></tr></tbody></table>

This holds for open-set recognition, but in few-shot there are additional complexities. Not only the classes from the query set may be unknown (i.e., not even in the support set), but the support set could provide irrelevant information for the current task. This setup, introduced in [175], is not common in the FSL literature even if it is of great interest from an industrial perspective. It is also quite relevant from the few-shot detection point of view as the detection support examples can embed irrelevant information for the task.

### Generalized Few-Shot Classification

Up to now, we presented the few-shot classification problem as only adapting a model to classify novel classes. However, it can sometimes be relevant to keep the possibility of classifying classes from the base dataset. Often the adaptation significantly reduces the performance on base classes, this phenomenon is known as the catastrophic forgetting [119]. When both base and novel classes are of interest, the task is called *generalized few-shot learning*. This can be achieved with several tricks such as doing the inference with both base-trained and fine-tuned models. But it is also possible with careful extension and fine-tuning of the model, e.g., via disentangling base and novel class predictions [176].

### Continual Learning

Generalized few-shot is an intermediary step toward continual or life-long learning which consists in continuously adapting the model with novel classes. This is way more challenging but also resembles the industrial setting better. While extremely relevant from COSE's perspective, we choose not to tackle this problem in this PhD project as it seems more sensible to address first the already challenging few-shot setting for the detection. In addition, continual learning often leverages complex learning scheme such as task rehearsal [177] or adaptive model architectures [178] to prevent forgetting classes or tasks.
### 2.2.2 Cross Domain Adaptation

Sometimes, there are significant discrepancies between images from train and test datasets. We discussed in the previous section the discrepancies in terms of classes: classes encountered at test time may differ from annotated training classes. However, training and test images can also have different aspects. For instance, autonomous vehicle perception systems could be trained only with daylight images and encounter nighttime images once deployed. The train and test image spaces are denoted as *source domain* and *target domain* in the Cross-Domain Adaptation (CDA) literature. Specifically, a domain consists in an image space $\mathcal{I}$ and a marginal probability distribution $p(I)$ over it:

$$ \mathcal{M} = \{\mathcal{I}, p(I)\}, \quad I \in \mathcal{I}. \qquad (2.22) $$

CDA aims at adapting a model trained for a specific task on a source domain $\mathcal{M}_{source} = \{\mathcal{I}_{source}, p_{source}(I)\}$ to perform the same (or another) task on the target domain $\mathcal{M}_{target} = \{\mathcal{I}_{target}, p_{target}(I)\}$. For simplicity, we restrain the scope of this section to the classification task. Hence, when the task changes from source and target, the set of classes changes as well. We denote these sets as $C_{source}$ and $C_{target}$ to comply with the CDA notations. Note that these sets of classes correspond to the base and novel classes in the FSC context. Generally, in the CDA literature, a limited amount of annotated data is available for the target domain which prevents direct supervised training. However, if a closely related source domain with sufficient available data is available, adaptation to the target domain is possible with limited data. Accordingly, cross-domain adaptation and few-shot learning are closely related problems. In this section, we review the two kinds of CDA, with and without label shift. COSE's industrial application contains CDA's problematics as the imaging systems can be deployed to different theaters of operations for which no images were available during training.

#### 2.2.2.1 Domain Adaptation without class shift

There exists a slight difference between Domain Adaptation (DA) and what is sometimes called Few-shot Domain Adaptation (FSDA) in the literature. This difference lies in the amount of available data in the target domain. FSDA methods have access to fewer target examples than regular DA. This distinction is not relevant as in both cases, there is not enough target data to perform directly supervised training (although additional unlabeled target data is often leveraged). Therefore, we choose to review both DA and FSDA at once. This review is not exhaustive, and we refer the reader to [179] for a more complete overview of Domain Adaptation. Following this survey, we divide our review into two parts, discrepancy-based adaptation and generative modeling approaches.

#### Discrepancy-based Adaptation

The simplest way to adapt a model to a target domain is to fine-tune it on the few available target data. The model is first trained on the source domain to learn the task. Then, fine-tuning is done on the target domain with some tricks to avoid overfitting. These tricks consist in reducing the discrepancies between source and target features. For instance, [180] fine-tunes on the target domain with
a regular cross-entropy loss but leverages additional loss functions to minimize domain confusion with additional unlabeled target images. Similarly, [180] has been extended with semi-supervised consistency [181] and contrastive [182] losses. Following the same principle, a number of works [183, 184, 185, 186] leverage additional losses based on the Maximum Mean Discrepancy (MMD) or close extensions. MMD is a distance measure between probability distributions. In the context of DA, it can be leveraged to assess the shift from source to target domain for a given class. Employing MMD-based loss functions allows these methods to learn domain invariant features and therefore improve cross-domain generalization. As an example, Central Moment Discrepancy (CMD) [187] proposes an approximation of MMD to derive a discrepancy regularizer. This regularization is computed over all layers of the model to enforce features from all levels to be domain invariant. Other contributions developed relatively similar techniques based on other criteria such as Kullback-Leiber divergence [188], or correlation alignment [189].

The methods presented above all fine-tune the models from feature discrepancies. However, as the task remains the same, it is reasonable to assume that optimal weights for the source and target domains are related. Following this idea, [190] proposes a weight regularization to prevent fine-tuning to find weights too different from source weights. Closely related, [191] proposes to only change Batch Normalization's statistics to adapt to the target domain.

Finally, advances in adversarial learning provided new ways to address DA by minimizing source and target discrepancies in an adversarial setup. This is embodied by [192] and [193] which both jointly train a domain discriminator along with the target feature extractor in an adversarial fashion. The trained extractor embeds images in a shared source-target feature space on which the source classifier can perform well.

### Generative Modeling

Another approach to domain adaptation is to artificially generate target data. This is particularly easy with discriminative approaches based on Generative Adversarial Networks [194]. GANs were extended to perform domain translation with CoGAN [195], Pix-2-Pix [196] and CycleGAN [197]. The source domain images can then be converted into source-target image pairs which greatly facilitate domain adaptation with methods similar to the ones described in the previous paragraph. This is done for instance in CyCADA [198]. Of course, GANs are not the only available generative models suitable for this task. Recent advances in image generation leveraging Diffusion Processes [199, 200] unveil new possibilities for domain adaptation following existing work about generative domain adaptation as done very recently by [201].

Closely related, Deep Reconstruction Networks (DRCN) [202] jointly learn to classify and reconstruct images from multiple domains. The model is trained to classify source images and reconstruct target images. This strategy enforces the learning of domain-invariant features and largely improves domain adaptation. Similar approaches have been proposed with disentangled domain-invariant and domain-specific representations [203], or adversarial reconstruction [204].
2.2.2.2 Cross-Domain Adaptation with class shift

Cross-Domain Few-Shot Classification (CD-FSC) designates problems where both classes and do-
main change at the same time. This complexifies further the learning, but it is closer to real-case
scenarios and developing such techniques will ease the deployment of classification techniques. It is
particularly interesting for COSE as it solves two major issues regarding training visual recognition
systems for surveillance applications: undefined objects of interest and changing image appearance.
This setting is relatively new in the few-shot literature and has been popularized in particular by
the creation of Meta-Dataset [205]. Meta-Dataset is a benchmark for CD-FSC. It gathers 10 existing
classification datasets and proposes a simple testing scenario: pre-train on ImageNet then fine-tune
on each dataset individually with limited annotations.

Most of the proposed techniques for solving CD-FSC borrow from both the few-shot learning and
domain adaptation fields. Plenty of approaches are then based on the meta-learning strategy, pre-
training on the source dataset and fine-tuning episodically on the target domain and novel classes.
Meta-FDMixup [206] for instance trains episodically a classifier with additional domain discrimi-
nant losses computed on an augmented query set (mixing-up source and target domain – MixUp
[207] is a well-known augmentation technique). Meta-FDMixup, is later extended with a dynamic
mixup strategy by Target Guided Dynamic Mixup (TGDM) [208]. Another merger of FSL and DA
techniques is Domain-Adaptive Prototypical Networks (DAPN) [209], which extends prototypical
networks with a domain adaptation module for prototype alignment, trained in an adversarial fash-
ion. Closely related, [210] proposes a bi-directional prototype alignment. Another line of work
tackles CD-FSC through the prism of distillation, for instance, [211] first trains two "experts" net-
works to perform the FSC task on both domains independently. Then, a student network is trained to
match the output of both teachers using distillation techniques. It results in a student network able
to deal with both domains identically. Similarly, Universal Representation Learning (URL) [212] dis-
tills knowledge learned from K classifiers trained on K distinct domains into a single cross-domain
model. This is achieved by adding lightweight domain adaptation modules between the feature ex-
traction module and the classification layer. Overall these techniques all involve complex training
strategies and architectural designs which are not very convenient for industrial deployment, repli-
cation, or future extensions. To counter this, ReFine [213] proposes a simple fine-tuning strategy
that only re-initializes the last layers of the model before fine-tuning to facilitate domain adaptation.
Much simpler than concurrent approaches, it yields competitive results.

Finally, some other works [214, 215, 216] study an even harder task when target domain data are
completely unlabeled. We will not review this kind of approach as it is out of the industrial scope
of COSE.
### 2.3 Conclusion

This chapter presents the object detection and few-shot learning fields, both necessary to the conception of few-shot object detectors. For object detection, notations and problem definitions are given in detail, as well as a list of popular evaluation metrics and datasets. A thorough review of existing works redraws decades of progress in this field and helps understand how state-of-the-art detection has been achieved. Similarly, for Few-Shot Learning, this chapter gives the key definitions to understand the stakes of the few-shot problem. An overview of the few-shot literature also provides relevant insights about how to adapt perception models in low-data regimes. This prospecting work greatly helps in understanding what is relevant from a research perspective and what directions to follow according to the industrial needs of COSE.
CHAPTER 3

# FEW-SHOT OBJECT DETECTION

## ABSTRACT

This chapter presents the task of detection in the few-shot regime and reviews the existing literature about it. Few-Shot Object Detection (FSOD) is at the crossroads of Object Detection and Few-Shot Learning, and therefore, extensively relies on these two fields explored in Chap. 2. Just as for classification, various directions are explored in the literature to tackle the detection task in the few-shot regime which will be presented in detail. Finally, this chapter focuses on the aerial image application of FSOD methods and extensions of the few-shot setting.

## CONTENTS

<table><tr><td>3.1 Problem definition</td><td>62</td></tr><tr><td>3.2 Review of the Few-Shot Object Detection Literature</td><td>63</td></tr><tr><td>3.2.1 Fine-tuning</td><td>63</td></tr><tr><td>3.2.2 Metric-learning based methods.</td><td>66</td></tr><tr><td>3.2.3 Attention-based methods.</td><td>67</td></tr><tr><td>3.2.4 Meta-Learning</td><td>70</td></tr><tr><td>3.3 Few-Shot Object Detection on Remote Sensing Images.</td><td>70</td></tr><tr><td>3.4 Extension of the Few-Shot Object Detection Setting</td><td>71</td></tr><tr><td>3.5 Dataset preparation and evaluation in the Few-Shot setting</td><td>73</td></tr><tr><td>3.5.1 Adapting detection datasets in the Few-Shot setting</td><td>73</td></tr><tr><td>3.5.2 Evaluation protocol for Few-Shot Object Detection</td><td>74</td></tr><tr><td>3.6 Conclusion</td><td>76</td></tr></table>

The company COSE is developing CAMELEON, an intelligent airborne surveillance system to automatically detect objects of interest. The detection algorithm must be adaptive as the objects can change from one operation to another. Therefore, the most relevant direction to explore is the
Few-Shot Object Detection (FSOD) task. In this chapter, we properly define the FSOD setting and present an exhaustive review of the current literature. We also explain how detection datasets can be leveraged for FSOD and how the proposed methods are evaluated.

## 3.1 Problem definition

Unsurprisingly, the Few-shot Object Detection task aims to detect objects just as regular object detection but under the few-shot constraints. Specifically, given an input image $I$, FSOD's goal is to learn a detection model $\mathcal{F}(\cdot, \theta)$, with parameters $\theta$, able to adapt to new classes ($C_{\text{novel}}$) from only a limited number of examples. Just as for the few-shot classification problem, two datasets are available, a base dataset with plenty on annotations of base classes instances $C_{\text{base}}$ and a novel dataset (also called support set) with $K$ annotated images for each novel class:

$$ \mathcal{D}_{\text{base}} = \{(I_i^c, \mathcal{Y}_i^c)\}_{1 \le i \le |\mathcal{D}_{\text{base}}|}, \quad c_i \in \mathcal{C}_{\text{base}}, \qquad (3.1) $$

$$ \mathcal{D}_{\text{novel}} = \bigcup_{c \in \mathcal{C}_{\text{novel}}} \{(I_k^c, \mathcal{Y}_i^c)\}_{1 \le k \le K}, \qquad (3.2) $$

where $I_k^c$ is an image containing at least one instance of the class $c$, and $\mathcal{Y}_k^c$ is the corresponding annotation set (bounding box and label) for the image $I_k^c$, filtered to contain only class $c$ instances. Note that there could be more than $K$ annotations per class as multiple objects of the same class can be visible on one image. This setting is commonly used in the FSOD literature and called N-ways K-shots object detection. Conversely, keeping only one annotation to comply with the few-shot classification setting can be problematic as it provides wrong supervision to the model. This issue will be elaborated further in Chap. 4. Hence, based on the input image and the support set, the few-shot detection model $\mathcal{F}(\cdot, \theta)$ should predict bounding boxes and labels for all instances of classes $C_{\text{novel}}$:

$$ \mathcal{F}(I, \mathcal{D}_{\text{novel}}) = \hat{\mathcal{Y}} = \{\hat{\mathbf{y}}_i\}_{i=1}^{M_I} = \{(\hat{b}_i, \hat{c}_i)\}_{i=1}^{M_I}, \quad \text{with } \hat{c}_i \in \mathcal{C}_{\text{novel}}. \qquad (3.3) $$

This setup resembles the FSC setting described in Chap. 2, but brings some complications. While the sets of base and novel classes are disjoint, FSOD must deal with the background. Any object that does not belong to either the base or novel class sets is considered background. Therefore, an object detector can encounter unknown classes at test time and must be able to ignore them. No information about the background classes is available which makes it even more difficult to discriminate between classes of interest and background. From this perspective, FSOD is closer to the few-shot open-set recognition problem than FSC. In addition, multiple different classes of interest can be depicted within a single image. Distinct objects can overlap in the image and their features (potentially from different classes) can blend, making recognition challenging. This is reinforced as the objects get smaller, their features get noisier and can be misclassified more easily. This stands for the query images but also for support images which increases the difficulty compared to FSC.
<figure><img src="image_50.png" /><figcaption>Figure 3.1: Timeline of the FSOD literature, several works are included as milestones for each of the four kinds of approaches to FSOD: Fine-tuning, Metric Learning, Meta-Learning and Attention-based approaches. The yellow hatched rectangle represents the duration of this PhD project.</figcaption></figure>

## 3.2 Review of the Few-Shot Object Detection Literature

Even though FSOD is a natural extension of FSC, the difficulties mentioned above prevent the direct use of FSC techniques, just as classification techniques may be extended for detection. Of course, the main principles for adapting classification models to the few-shot setting can be reused, but they need to be carefully adjusted to take care of the supplementary challenges of the detection task. Hence, as for FSC, the detection models are first trained on the base dataset and then adapted to novel classes with the support set. This adaptation can be done in many ways, often based on FSC approaches. Therefore, we adopt the same organization as for Sec. 2.2 and divide our review into four distinct parts: fine-tuning, metric learning, meta-learning and attention-based approaches.
Fig. 3.1 outlines the organization and the temporality of the FSOD field. FSOD is a relatively new challenge and only started 2 years before this PhD project. Tab. 3.1 provides an almost exhaustive overview of the literature about FSOD. The reader can refer to several surveys [217, 218, 219] about FSOD for more thorough reviews. However, note that these surveys are already a few years old, which is already a lot compared to the recency of the field.

### 3.2.1 Fine-tuning

Fine-tuning is the simplest approach to tackle FSOD, the principle is quite straightforward and similar to FSC: train a detection model to detect base classes on a large dataset and then fine-tune it