3. Freeze the entire backbone and let only the detection head be updatable. This choice is not optimal and will be discussed in the next section, however, we present here the baseline configuration.

4. Reset the learning rate scheduler, so it goes again through a warmup phase. The scheduling is a simple linear warmup starting at $\frac{1}{1000}$ of the base Learning Rate (LR) and linearly increasing up to the base LR value during 1000 iterations.

5. Fine-tune the model with *K* images for each novel class. All instances of the novel classes are kept while instances from other classes are discarded. This corresponds to the distractor-free sampling scheme discussed in Sec. 4.1.

As our goal is not to tackle the Generalized Few-Shot setting, we are mostly interested in the performance on novel classes. Of course, one might want to detect base classes as well, in this case, it is possible to keep a version of the model after base training and leverage it for base classes detection. Of course, it would require twice as much time to perform the inference to detect both base and novel classes, but this is a mild compromise compared to common issues raised by G-FSOD.

Fine-tuning is part of most FSOD methods as the adaptation of the regression part of the models cannot be easily done on the fly (conversely to the classification part). However, fine-tuning attention-based or metric learning models is often quite long in comparison with "simple" fine-tuning strategies which directly fine-tune object detectors on the support set without expensive additional components (e.g., a query-support attention block). This makes the fine-tuning faster and unlocks much quicker iterations and experiments. Nevertheless, fine-tuning approaches cannot be adapted at inference time and therefore, it is difficult to measure the robustness to various support examples. Thus, multiple fine-tunings are required to get a relevant evaluation of a model, otherwise, the randomness of the support set can introduce some variance and the comparison is less reliable. In practice, fine-tuning with different support does not significantly change the performance of FSDiffusionDet.

While the proposed strategy is fairly simple, it yields impressive results. We provide in Tab. 7.1 a comparison between the FSDiffusionDet baseline and the discussed methods from previous chapters. It outperforms largely the metric-learning and attention-based methods on aerial images. On natural images, the gains are reduced but FSDiffusionDet is still superior (especially for MS COCO where the problem is now a 20-ways detection problem and not a 5-ways task as in previous chapters). A detailed analysis is conducted in the next section to understand why it performs so well and how it can be improved further.

Another advantage of FSDiffusionDet compared with attention methods is its memory efficiency. Indeed, query-support combination blocks and support embedding models require a lot of memory while training and often scale linearly with the number of classes (*N*) and the number of shots *K*. FSDiffusionDet is not limited by the number of shots and therefore, we can explore much higher shot settings than with metric-learning or attention-based methods. Tab. 7.2 provides the novel classes
<table><caption>Table 7.1: FSDiffusionDet baseline compared with other FSOD methods. mAP is reported with a 0.5 IoU threshold and all methods leverage 10 shots.</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">DOTA</th><th colspan="2">DIOR</th><th colspan="2">Pascal VOC</th><th colspan="2">MS COCO</th></tr><tr><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th><th>Base</th><th>Novel</th></tr></thead><tbody><tr><td>FRW</td><td>49.04</td><td>35.29</td><td>61.30</td><td>37.29</td><td>63.21</td><td>48.72</td><td>29.03</td><td>24.09</td></tr><tr><td>DANA</td><td>53.99</td><td>36.50</td><td>62.71</td><td>38.18</td><td>65.17</td><td>52.26</td><td>38.14</td><td>24.75</td></tr><tr><td>WSAAN</td><td>46.72</td><td>35.12</td><td>62.79</td><td>32.38</td><td>65.27</td><td>51.70</td><td>40.87</td><td>21.42</td></tr><tr><td>PFRCNN</td><td>36.32</td><td>11.55</td><td>42.37</td><td>9.16</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>XQSA</td><td>51.11</td><td>41.00</td><td>59.88</td><td>41.51</td><td>62.13</td><td>53.94</td><td>31.56</td><td>25.03</td></tr><tr><td>FSDiffDet</td><td>69.58</td><td>52.05</td><td>81.71</td><td>54.32</td><td>74.63</td><td>52.64</td><td>51.91</td><td>24.99</td></tr></tbody></table>

<table><caption>Table 7.2: Influence of the number of shots on the few-shot object detection performance of FSDiffusionDet on DOTA, DIOR, Pascal VOC and MS COCO. Performance is reported with mAP<sub>0.5</sub>.</caption><thead><tr><th>K</th><th>DOTA</th><th>DIOR</th><th>Pascal VOC</th><th>MS COCO</th></tr></thead><tbody><tr><td>1</td><td>4.19</td><td>27.17</td><td>22.24</td><td>7.43</td></tr><tr><td>2</td><td>9.83</td><td>40.31</td><td>31.98</td><td>12.45</td></tr><tr><td>3</td><td>27.61</td><td>43.54</td><td>29.52</td><td>15.75</td></tr><tr><td>5</td><td>39.00</td><td>46.92</td><td>38.08</td><td>19.33</td></tr><tr><td>10</td><td>52.05</td><td>54.32</td><td>52.64</td><td>24.99</td></tr><tr><td>20</td><td>62.79</td><td>60.24</td><td>59.26</td><td>28.76</td></tr><tr><td>30</td><td>67.32</td><td>65.28</td><td>64.19</td><td>31.19</td></tr><tr><td>50</td><td>71.91</td><td>71.21</td><td>67.81</td><td>34.64</td></tr><tr><td>100</td><td>72.27</td><td>77.05</td><td>71.31</td><td>38.77</td></tr></tbody></table>

performance on our four datasets of interest. The base class performance is not reported here as they do not depend on the number of shots, they can be found in the last row of Tab. 7.1. One can observe a smooth increase in performance with the number of shots with a plateau above 50 shots. To better visualize this trend and compare it with attention-based methods studied in Chap. 6, we plot in Fig. 7.5 the performance against the number of shots. From this, it can be seen that the performance is much lower in the one-shot setting with FSDiffusionDet compared to attention-based approaches. However, FSDiffusionDet quickly catches up and outperforms largely other methods in higher shots settings. In addition, we can observe a much quicker increase in performance as the number of shots increases with FSDiffusionDet. This is a highly desirable property in an industrial application because this means that the model has more potential for improvements. On the contrary, attention-based approaches do not display such a strong trend, they are better suited for extremely low-shot regimes, but become less effective with higher shots.

### 7.3.2 Experimental Study of FSDiffusionDet

In the previous section, we have presented an efficient fine-tuning strategy for DiffusionDet along with an analysis of its few-shot performance on several datasets. However, this is only a baseline for FSDiffusionDet and its performance can be improved further. Its fast training time allows for
<figure><img src="image_88.png" /><figcaption>Figure 7.5: Performance of FSDiffusionDet, XQSA, FRW, DANA and WSAAN on DOTA, DIOR, Pascal VOC and MS COCO against the number of shots. Performance is reported with mAP<sub>0.5</sub>.</figcaption></figure>
<table><caption>Table 7.3: Study of the influence of the backbone pre-training. mAP<sub>0.5</sub> is provided only for base classes, therefore the blue and red colors to distinguish between base and novel classes are no longer required.</caption><thead><tr><th>Backbones</th><th>DOTA</th><th>DIOR</th><th>Pascal VOC</th><th>MS COCO</th></tr></thead><tbody><tr><td>Scratch</td><td>7.28</td><td>8.72</td><td>13.72</td><td>0.38</td></tr><tr><td>ImageNet</td><td><strong>52.05</strong></td><td>54.32</td><td>52.64</td><td><strong>24.99</strong></td></tr><tr><td>DINO</td><td>46.84</td><td><strong>55.88</strong></td><td><strong>54.58</strong></td><td>23.94</td></tr><tr><td>CLIP</td><td>40.36</td><td>51.61</td><td>49.81</td><td>19.83</td></tr></tbody></table>

conducting more experiments, which was too expensive with metric-learning and attention-based methods. Thus, we present in this section a series of experiments that we conducted to explore the capabilities of FSDiffusionDet but also to answer more general questions about FSOD.

### 7.3.2.1 Backbone Weights Initialization

First, in classification, it is now well-known that using self-supervised pre-trained backbones often boost a lot the few-shot performance of a method [113, 332, 68, 291]. While plenty of studies show this for classification, for detection, the transferability of the learned features is not so evident. As a matter of fact, a few contributions actually show that using such backbones is sub-optimal. For instance, InsLoc [333] and SoCO [334] propose object-level self-supervised techniques and prove empirically that image-level SSL is not optimal and in some cases can even be detrimental to the detection task. In this section, we study empirically the influence of using SSL pre-trained weights for the backbone initialization, before base training and fine-tuning. We do not consider the recent object-level techniques and instead leverage four different initialization strategies for the backbone weights:

- **Scratch**: weights are randomly initialized.

- **ImageNet**: weights are initialized from a ResNet-50 trained in a supervised manner for ImageNet classification.

- **DINO**: weights from the DINO [68] pretraining on ImageNet.

- **CLIP**: weights taken from the CLIP [291] model, trained in a contrastive way on a 400 million image-text pairs dataset.

Then, FSDiffusionDet is trained following our two steps training scheme (base training and fine-tuning). The results are available in Tab. 7.3. From this, it is quite clear that training from scratch is not a sensible option, even though base training is correct, the fine-tuning on base classes provides really poor detection performance. Then, between Imagenet, DINO and CLIP the differences are tight. Of course, CLIP's weights are slightly worse than ImageNet and DINO, but it is still a strong baseline. Between ImageNet and DINO, however, it is difficult to conclude as both achieve the best performance on one aerial and one natural dataset. As the performance gap is thin between ImageNet and DINO, we choose to conduct our next experiments with ImageNet weights which have stood the test of time and are now the default choice in computer vision.
### 7.3.2.2 Plasticity Analysis

The number of parameters frozen in the model is sometimes called the *plasticity* of the model in the continual learning field, but this concept may also be useful in the few-shot setting. For simplicity, we measure the plasticity of the model as the ratio between the number of trainable parameters over the total number of parameters. Plasticity close to 1 means that the model is malleable and could learn new complex tasks. However, when it is close to 0, the model can barely change and learning new tasks may be difficult.

The plasticity is commonly explored in fine-tuning strategies for few-shot tasks. The underlying principle is that the task is learned during base training, and fine-tuning is only used to adapt the task to novel classes. Hence, the behavior of the entire model should not change dramatically. Therefore, the plasticity of the models is often quite low in the FS literature. In practice, the early stages of the model are kept frozen while only the deeper layers are trained. This trick is well-motivated as it drastically reduces the capacity of the model and thus prevents overfitting, which is particularly severe in low-shot regimes. In addition, it also reduces catastrophic forgetting, which can be quite a challenge in G-FSOD.

However, this may be inadequate for the detection task. As the detection is primarily a problem of finding what is and what is not an object of interest, the backbone is trained as a feature filter. Features from classes of interest are highlighted while others are faded out. Conversely, for classification, backbones are not required to learn such a filtering process as all classes are "of interest", and there is always one object of interest in the image. In addition, some recent experiments [284] about freezing settings in cross-domain scenarios show that improved performance is achieved with increased plasticity. In this work, the authors only study three freezing settings: fine-tuning only the last layer (as proposed in TFA [258]), fine-tuning only the detection head (proposed in FSCE [254]), and fine-tuning the whole model (their proposition). Here, we investigate the freezing setting in a more detailed manner with several intermediary setups, but the main difference is that we conduct this analysis on the same image domain:

- **Fine-Tune last layer only**: fine-tune only the last regression and classification layers.

- **Fine-Tune head only**: fine-tune only the detection head.

- **Up to stage *i***: Freeze backbone up to stage number *i* ($i \in [1, 5]$ as ResNets have 5 stages).

- **Fine-Tune whole**: fine-tune the whole model.

- **Bias only**: fine-tune only the backbone biases.

- **BatchNorm only**: fine-tune only the backbone BatchNorm parameters.

The results of this comparison can be found in Tab. 7.4. It reports the mAP with different freezing strategies on DOTA, DIOR, Pascal VOC and MS COCO. Additionally, the plasticity rate is reported for each freezing strategy. Two distinct behaviors are observed here. First, on DOTA, as the plasticity increases, the FSOD performance increases as well. On DIOR, Pascal VOC and MS COCO, lower plasticity is optimal (fine-tuning the detection head and the last stage of the backbone). Therefore, the fine-tuning strategy cannot be set once for all datasets. It is therefore crucial to understand what
<table><caption>Table 7.4: Influence of the amount of plasticity on the FS performance on DOTA, DIOR, Pascal VOC and MS COCO. mAP is reported with a 0.5 IoU threshold.</caption><thead><tr><th>Freezing point</th><th>Plasticity rate</th><th>DOTA</th><th>DIOR</th><th>Pascal VOC</th><th>MS COCO</th></tr></thead><tbody><tr><td>FT whole</td><td>100.00 %</td><td>60.09</td><td>52.17</td><td>43.10</td><td>17.15</td></tr><tr><td>Bias only</td><td>35.98 %</td><td><strong>60.45</strong></td><td>55.12</td><td>49.90</td><td>20.19</td></tr><tr><td>BatchNorm only</td><td>35.97 %</td><td>59.35</td><td>55.63</td><td>51.96</td><td>19.70</td></tr><tr><td>Up to stage 1</td><td>99.98 %</td><td>58.85</td><td>53.37</td><td>43.81</td><td>17.72</td></tr><tr><td>Up to stage 2</td><td>99.47 %</td><td>57.41</td><td>53.21</td><td>41.23</td><td>17.73</td></tr><tr><td>Up to stage 3</td><td>96.57 %</td><td>59.88</td><td>54.36</td><td>47.57</td><td>19.49</td></tr><tr><td>Up to stage 4</td><td>79.66 %</td><td>56.13</td><td><strong>57.51</strong></td><td><strong>53.72</strong></td><td><strong>21.88</strong></td></tr><tr><td>FT head only</td><td>35.97 %</td><td>51.82</td><td>55.70</td><td>51.72</td><td>19.96</td></tr><tr><td>FT last layer only</td><td>0.03 %</td><td>0.05</td><td>0.11</td><td>0.53</td><td>0.01</td></tr></tbody></table>

differs in DOTA from the other datasets. As the task and the images remain similar between base training and fine-tuning, the only source of variability comes from the class splits. Our hypothesis here is that base and novel classes in DOTA are less compatible (i.e., less alike) than in the other datasets. For Pascal VOC, we briefly discuss this aspect in Sec. 6.2.5.2, where we observed surprisingly high mAP for the novel class *sheep* as the class *horse* was in the base set. A more quantitative way of measuring the compatibility between the base and novel class sets would be required to draw reliable conclusions about this. We are currently working on this.

In addition, we observe that fine-tuning the backbone entirely is often a sub-optimal choice. Instead, higher (or at least competitive) results are achieved by fine-tuning only the biases or the batch normalization parameters of the backbone. Fine-tuning only the biases or the batch normalization parameters in the backbone does not change much the plasticity as only a few parameters are concerned, yet it seems to provide a beneficial adaptability to the entire backbone. On DIOR, Pascal VOC and MS COCO, it provides very high mAP compared to other settings with similar plasticity. Finally, fine-tuning only the very last layer of the classification and regression branches is completely sub-optimal. Strangely, this contradicts some FSOD models that adopt this strategy and achieve reasonable performance (e.g., TFA [258]). With FSDiffusionDet, this strategy achieves extremely poor detection, having too small plasticity must be avoided. Thus, a plasticity compromise must be found depending on the dataset and its split compatibility.

### 7.3.2.3 Number of Proposals

Another set of experiments explores the influence of the number of *proposals* for FSOD. The proposals are the boxes sampled at the beginning of the diffusion process. The number of proposals $N_p$ represents the maximum number of objects that the model can detect in one image. This number is chosen large compared to the average number of objects in the images. Intuitively, sampling more random boxes reduces the chances of missing an object. However, having a higher number of proposals generates more duplicates which can be detrimental as well. More proposals also lead to a higher training time and memory usage as the denoising process is applied on all boxes. The critical
<table><caption>Table 7.5: Analysis of FSDiffusionDet performance (mAP<sub>0.5</sub>) against the number of proposals on DOTA, DIOR, Pascal VOC and MS COCO datasets.</caption><thead><tr><th># of Proposals</th><th>DOTA</th><th>DIOR</th><th>Pascal VOC</th><th>MS COCO</th></tr></thead><tbody><tr><td><strong>200</strong></td><td>41.57</td><td>52.92</td><td><strong>52.86</strong></td><td><strong>23.24</strong></td></tr><tr><td><strong>250</strong></td><td>47.97</td><td>47.62</td><td>52.28</td><td>22.61</td></tr><tr><td><strong>300</strong></td><td><strong>55.76</strong></td><td>51.77</td><td>51.81</td><td>22.46</td></tr><tr><td><strong>350</strong></td><td>52.27</td><td>50.41</td><td>50.63</td><td>22.13</td></tr><tr><td><strong>400</strong></td><td>46.49</td><td>49.98</td><td>50.55</td><td>20.04</td></tr><tr><td><strong>450</strong></td><td>53.11</td><td>53.07</td><td>51.06</td><td>20.48</td></tr><tr><td><strong>500</strong></td><td>52.03</td><td><strong>55.31</strong></td><td>51.44</td><td>20.25</td></tr></tbody></table>

parts are the self-attention layers that scale in $O(N_p^2)$. Thus, we investigate the few-shot performance of FSDiffusionDet with various numbers of proposals. The results of these experiments are available in Tab. 7.5. We notice two different behaviors between natural and aerial images. For natural images (Pascal VOC and MS COCO), it seems better to set the number of proposals relatively low compared to aerial images. This makes sense as there are more objects in aerial images. For natural images, the detection quality increases as the number of proposals is reduced, and it may be relevant to test what happens with even fewer proposals. However, with aerial images, the performance does not seem to correlate well with the number of proposals. It is relevant to mention that the results on MS COCO are opposite to what the authors of DiffusionDet found in the regular data regime (increasing the number of proposals increases the mAP). This could be explained by the reduced number of objects in the images, as in the few-shot regime we consider only the novel classes, many instances are discarded and fewer proposals are required to detect the objects.

### 7.3.2.4 Other Experiments and Future Directions

In addition to the previous experiments, we conduct several other studies to further improve the detection capabilities of FSDiffusionDet. However, some of these studies did not yield very relevant insights, some others were not explored deeply enough due to the time constraint of this PhD project. We briefly present these experiments that will pave the way for future improvements of FSDiffusionDet.

#### Learning Rate Sweeping.

First, just as proposals, freezing sweet spot and backbone pre-training, we studied the influence of the Learning Rate (LR) and its schedule on the FSOD performance. Indeed, the choice of the LR value during the fine-tuning is not trivial. Therefore, to make sure we get a good fit for our experiments we conduct a LR sweeping, i.e., we try several different values for the LR. This experiment is conducted only on DOTA with $K = 10$ for simplicity, but in theory, it should be done for every new experiment. Indeed, following the nomenclature from the recent Deep Learning Tuning Playbook [335], the learning rate is a nuisance parameter, meaning that to make a fair comparison between various settings, the optimal LR should be found for all runs individually. As we change some hy-
<table><caption>Table 7.6: Learning rate sweeping on DOTA dataset with $K = 10$ shots. Two distinct schedulers are considered: constant and cosine annealing. Performance is reported with mAP<sub>0.5</sub>.</caption><thead><tr><th>Learning rate</th><th>Constant Schedule</th><th>Cosine Annealing</th></tr></thead><tbody><tr><td>1e-6</td><td>39.04</td><td>29.45</td></tr><tr><td>5e-6</td><td>49.06</td><td>45.32</td></tr><tr><td>1e-5</td><td>52.31</td><td>49.33</td></tr><tr><td>5e-5</td><td><strong>53.46</strong></td><td><strong>52.99</strong></td></tr><tr><td>1e-4</td><td>53.25</td><td>51.96</td></tr><tr><td>5e-4</td><td>49.33</td><td>52.51</td></tr><tr><td>1e-3</td><td>47.45</td><td>47.82</td></tr></tbody></table>

perparameters, it is likely that the optimal learning rate changes as well, therefore fair comparison can only be achieved if the LR is optimal for all runs, e.g., the optimal learning rate for $K = 1$ or $K = 100$ shots may not be the same. Even though fine-tuning methods are fast to adapt, running such an LR analysis is very expensive. Nevertheless, running an LR sweeping on DOTA provides insights into how it influences the FSOD performance. The results can be found in Tab. 7.6, and show an optimal value around $5e-5$. But most importantly, it shows a relatively large area where performance is satisfactory. This comforts us in our choice of fixing the LR for all our experiments. While this is probably not the optimal choice, it is reasonable. We also tried a cosine annealing scheduler, but it yields consistently inferior results and was then rejected. Its only advantage is that it seems to deal better with higher LR, which makes the training slightly faster.

## Proposal Prior Distribution.

In DiffusionDet, the coordinates of the proposals are sampled randomly following a normal distribution. The coordinates of the boxes are clamped with a scale parameter $\zeta$ to make sure the center of each box remains within the image limits. Specifically, we have:

$$w = (\text{clamp}(\epsilon_w, -\zeta, \zeta)/\zeta + 1)/2, \qquad (7.27)$$

$$h = (\text{clamp}(\epsilon_h, -\zeta, \zeta)/\zeta + 1)/2, \qquad (7.28)$$

$$x = (\text{clamp}(\epsilon_x, -\zeta, \zeta)/\zeta + 1)/2 - \frac{w}{2}, \qquad (7.29)$$

$$y = (\text{clamp}(\epsilon_y, -\zeta, \zeta)/\zeta + 1)/2 - \frac{h}{2}, \qquad (7.30)$$

$$\text{with } \epsilon_x, \epsilon_y, \epsilon_w, \epsilon_h \sim \mathcal{N}(0, 1). \qquad (7.31)$$

By default, $\zeta$ is set to 2, but it would be interesting to explore how it changes the FSOD performance. Indeed, as $\zeta \to 0$ the boxes are more and more identical and their centers tend to approach the image corners, as $\zeta \to \infty$, the boxes are more and more aligned with the image center. It could also be relevant to explore the use of a uniform sampling instead of a Gaussian distribution. This would prevent having a bias toward the image center as it happens with high values of $\zeta$. In fact, the use of $\zeta$ close to 1, is relatively close to a uniform distribution. This is illustrated in Fig. 7.6.
The setting of $\zeta$ slightly changes the diffusion process as the boxes are generated from clamped Gaussian distributions and not regular Gaussians. Most of the derivations detailed in Sec. 7.1 hold only for gaussian distributions and therefore using small values of $\zeta$ or uniform prior may disrupt the diffusion process. We did not have time to conduct these experiments yet, but this is planned as future work. Another consideration is the size of the generated boxes. In the above, the size of the proposal is randomly sampled and has an expected value of half the image size. This may not be optimal, especially when applying FSDiffusionDet on aerial images with small objects. Hence, we propose to introduce a proposal scaling parameter $\bar{\omega}$ that divides the width and height of the proposals:

$$w' = \frac{w}{\bar{\omega}}, \qquad (7.32)$$

$$h' = \frac{h}{\bar{\omega}}, \qquad (7.33)$$

where $w'$ and $h'$ are the scaled width and height of the sampled boxes. Of course, as small proposals may not cover the whole image, their number must be increased to prevent missing objects. Going further, we can also imagine a mixture of width and height distribution to sample proposals with significant size differences, which is not achieved in practice yet as shown in Fig. 7.6. Fig. 7.7 shows how the proposals would change with $\bar{\omega}$, with $\zeta = 2$ fixed.

## Transductive Inference.

To further improve the FSOD performance of FSDiffusionDet, we also consider designing a transductive inference scheme inside the detection framework. To our knowledge, this would be a first in the FSOD domain. Of course, the transductive setting is slightly different from the regular few-shot inference as it requires access to a large set of query images during the inference. The goal is to detect objects in these images, but these unlabelled images can be leveraged to improve the detection on the entire set. This setup makes a lot of sense for COSE's application. Indeed, the very large images of COSE cannot be processed as a whole, instead, they must be cropped into smaller patches. This means that a relatively large number of images are to be processed at the same time (11600 × 8700 pixels images can be cropped into roughly 400 patches of 512 × 512 pixels). Hence, studying transductive inference is of particular interest to the company.

First of all, in classification, the transductive inference is often used in replacement of a fine-tuning step and allows for direct adaptation of a model to a new task or domain (see Sec. 2.2.1.6 for more details). This is prohibited in the detection context as the regression branch must be fine-tuned anyway. Thus, our goal is to improve the classification part once the model has been fine-tuned. To this end, we propose to adapt LaplacianShot [162] to work with representations of objects instead of representations of entire images. Specifically, the Laplacian Shot Module (LSM) replaces the classification layer of the model. As input, LSM receives a set of all objects representation detected in the query set (using the boxes produced by the regression branch) and the representation of the annotated support examples. Then, LSM optimizes an objective function to find an optimal label
<figure><img src="image_89.png" /><figcaption>Figure 7.6: DiffusionDet initial random boxes with various values of $\zeta$, the parameter that controls the spread over the images. 75 proposals are sampled per image.</figcaption></figure>

<figure><img src="image_90.png" /><figcaption>Figure 7.7: Influence of $\bar{\omega}$ on the size of the proposals. Note that here 200 proposals are sampled, for visualization purposes.</figcaption></figure>
assignment $\mathcal{Y}$ of all query examples (in our case all objects found in the query set):

$$ \mathcal{L}_{\text{LSM}}(\mathcal{Y}) = \sum_{i=1}^{|Q|} \sum_{c=1}^{|C_{\text{novel}}|} l_i^c d(z_i, m_c) + \frac{1}{2} \sum_{i,j} \eta(z_i, z_j) l_i^T l_j, \qquad (7.34) $$

where $z_q$ and $l_q$ are the extracted features and classification score vector for object $q$, respectively. $Q$ represents the set of object representations in the whole query set, $d$ is a distance measure (e.g., the euclidean distance between objects representations), and $\eta$ is a similarity function in the embedding space (in practice, it is chosen as a binary $k$-NN, i.e., a vector has a similarity of 1 with its $k$ nearest neighbors and 0 with all others). Finally, $m_c$ is the representation of class $c$, it is computed as the average over the support representations of that class. Intuitively, this objective function finds a compromise between assigning to an unlabelled object the label of the closest support example and assigning the same label as its neighbors.

As a first comparison, we leverage four distinct inference setups for detection:

- **Fine-tuning Inference (FI):** boxes and classification scores output by the fine-tuned model.

- **Transductive Inference (TI):** boxes from fine-tuned model and classification score from the Laplacian Shot Module.

- **Hybrid Transductive Inference (HTI):** boxes from fine-tuned model and classification scores as a combination (e.g., element-wise multiplication) of LSM and fine-tuned model.

- **Optimal Classification (OC):** boxes from the model and optimally matched labels from the ground truth. It can be seen as an **oracle**, it is a performance upper-bound given the quality of the regression.

We assume above that for the detection task, the regression branch must be fine-tuned otherwise performance is highly degraded. To confirm this assumption, we compare the four inference settings described above using a model that has only been base-trained against a model that was fine-tuned on the novel classes. This is done on DOTA with $K = 10$ shots. The results are available in Tab. 7.7. From that table, it is clear that the fine-tuning of the regression branch is crucial to achieving reasonable performance. In particular, the oracle (OC) is highly degraded when the regression branch is not fine-tuned. Interestingly, in this case, TI achieves higher mAP than the non-fine-tuned classification branch of the model. It makes sense as this layer is only initialized with random weights. However, one can see that the TI is largely under the FI when using the fine-tuned regression branch. The classification made by the LSM is therefore worse than the fine-tuned classification branch. A quick investigation of the classification scores shows that the scores output by TI differ significantly from FI (see Fig. 7.9). The fine-tuned model outputs a large number of very small classification scores which mostly correspond to background objects. Hence, they are filtered out by the post-processing (score thresholding and NMS), and the remaining ones will have a negligible impact on the mAP computation. TI, however, outputs much higher scores, with a greater variance. It struggles to distinguish foreground and background objects, and for good reason, it was designed for classification
and not detection. To this end, we propose HTI, a hybrid classification inference that leverages both the score from the fine-tuned model and the transductive inference. Hopefully, it will fix the mistakes from the fine-tuned classification layer while avoiding the TI's pitfall. To do so, we simply multiply the scores from FI and TI together. Thus, the good foreground/background distinction from FI is embedded in the new score distribution. This helps a lot for the classification; however, it is still under the FI performance. From this, it seems that TI is only detrimental to classification. TSNE representations of the embedding space help to make sense of these results. These can be found in Fig. 7.8 for FI, TI and HTI, and the Oracle.

One can see large patches of the cluster representing the class 2 misclassified as class 14 by both TI and HTI. The prototype of class 2 seems closer to these misclassified points in the TSNE visualizations, but the distances must be interpreted carefully as two dimensions are not enough to represent the entire complexity of the representation space (dimension 256). Yet, it seems that the distributions of the classes in the embedding space are multimodal, therefore it may be impossible to accurately classify the objects with only one prototype per class. Leveraging multiple prototypes per class should be investigated in future work.

In addition to HTI, we also tried to filter the objects with low FI scores (with a threshold at 0.05), this greatly helps for the transductive inference although it reduces slightly the performance of FI (some objects are correctly detected but have a low score but are filtered anyway). With filtering and HTI, we almost reach the same performance as the FI which is encouraging. However, the goal is to benefit from the transductive inference and this is not achieved yet. More analysis needs to be conducted to better understand the reason behind the poor classification score of TI. In addition, the transductive inference should be extended to account for the detection challenges: a great foreground/background imbalance and an increased intra-class diversity. To this end, we have a few ideas that we did not have time to explore yet:

- **Geometric priors:** leverage the geometrical features of the objects (e.g., size, aspect ratio, etc) to find outliers and exclude them without using a hard score thresholding. This could significantly reduce the number of objects and help to filter ill-formed boxes.

- **Multiple prototypes per class:** instead of aggregating all support examples of one class as one prototype, use multiple prototypes per class and extend LSM as a mixture model.

- **Background class:** Introduce a background class within the LSM module to prevent the poor foreground/background distinction.

These tracks will be explored during the last months of this PhD project and will hopefully improve further the detection capabilities of FSDiffusionDet. In addition, we plan to apply the transductive inference in Cross-Domain scenarios, as it could help against the performance drop in these settings.

## Support Attention.

Finally, we also try to extend FSDiffusionDet with an attention mechanism (e.g., XQSA). The main motivation is to be able to compare the influence of the detection framework on attention-based
<figure><img src="image_91.png" /><figcaption>Figure 7.8: Comparisons of the TSNE visualizations of the fine-tuned model predictions (FI), trans-ductive inference (TI), hybrid inference (HTI) and the oracle. Note that background predictions are only available for the oracle as the models’ inferences only provide class scores.</figcaption></figure>
<figure><img src="image_92.png" /><figcaption>Figure 7.9: Comparisons of the scores histograms of fine-tuned model predictions (FI), transductive inference (TI), and hybrid inference (HTI).</figcaption></figure>

<table><caption>Table 7.7: Naive comparison between a model only trained on base classes against a model that has been fine-tuned as well. All 4 inference setups are compared as well. Bold values represent the best-performing method between FI, TI and HTI, the Oracle (OC) is not included. Performance is reported as the mAP<sub>0.5</sub></caption><thead><tr><th>Training Strategy</th><th>FI</th><th>TI</th><th>HTI</th><th>OC</th></tr></thead><tbody><tr><td>Base training only</td><td>0.04</td><td><strong>3.79</strong></td><td>2.56</td><td>25.42</td></tr><tr><td>Base training + FT</td><td><strong>58.98</strong></td><td>33.61</td><td>53.90</td><td>67.35</td></tr><tr><td>Base training + FT + Filtering</td><td><strong>57.95</strong></td><td>43.63</td><td>57.00</td><td>67.35</td></tr></tbody></table>

FSOD methods (in Chap. 6, we studied the influence of the attention mechanism with a fixed detection framework). Given the impressive results of FSDiffusionDet with a simple fine-tuning strategy, this is promising.

Thus, we extend the detection head with a query-support block which is meant to incorporate the support features within the detection head. The head is then split and boxes are produced for each class independently (following the attention-based FSOD principle, see Fig. 6.1). Unfortunately, this does not yield satisfactory results and slows down the training a lot. Considering the time already spent on attention mechanisms since the beginning of the project and the very good performance of FSDiffusionDet with the fine-tuning, we decided not to explore this direction further. Nonetheless, this remains an interesting direction for future work.

### 7.3.3 Comparison with existing FSOD Methods

In the previous section, we explored several design choices for FSDiffusionDet and analyzed how they influence the detection performance on novel classes. We compare here the best settings for FSDiffusionDet according to our experiments conducted in the previous section. These experiments are averaged over 5 distinct seeds to get more reliable results. This contrasts with the above experiments which are mostly done with one seed only. However, the limited variance observed over the multiple runs confirms that previous results are reliable as well. FSDiffusionDet is compared with PFRCNN and XQSA that we proposed in Chaps. 5 and 6 and some relevant works from the literature.
<table><caption>Table 7.8: Detection results of FSDiffusionDet on DOTA, DIOR, Pascal VOC and MS COCO datasets. The models employed to produce this figure have been finetuned with $K = 10$ shots and following the best fine-tuning strategy found in Sec. 7.3.2.4 for each dataset. The mAP$_{0.5}$ is reported as an average of over 5 distinct runs. The top rows include methods from the literature while the bottom rows designate our proposed methods.</caption><thead><tr><th rowspan="2"></th><th colspan="4">DOTA</th><th colspan="4">DIOR</th><th colspan="4">Pascal VOC</th><th colspan="4">MS COCO</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td>FRW</td><td>35.29</td><td>13.99</td><td>34.11</td><td>59.31</td><td>37.29</td><td>2.48</td><td>33.74</td><td>59.38</td><td>48.72</td><td>16.44</td><td>26.71</td><td>68.27</td><td>24.09</td><td>11.53</td><td>22.45</td><td><strong>38.69</strong></td></tr><tr><td>DANA</td><td>36.50</td><td>14.32</td><td>40.28</td><td><strong>64.65</strong></td><td>38.18</td><td>3.21</td><td>34.91</td><td>60.99</td><td>52.26</td><td>10.05</td><td>24.67</td><td>67.23</td><td>24.75</td><td>12.01</td><td>29.40</td><td>37.95</td></tr><tr><td>WSAAN</td><td>35.12</td><td>-</td><td>-</td><td>-</td><td>32.38</td><td>-</td><td>-</td><td>-</td><td>51.70</td><td>-</td><td>-</td><td>-</td><td>21.42</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PFRCNN</td><td>11.55</td><td>-</td><td>-</td><td>-</td><td>9.16</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>XQSA</td><td>41.00</td><td>17.84</td><td>44.57</td><td>54.46</td><td>41.51</td><td>4.12</td><td>40.69</td><td>58.21</td><td>53.94</td><td><strong>19.46</strong></td><td><strong>34.86</strong></td><td>66.14</td><td><strong>25.03</strong></td><td><strong>12.57</strong></td><td><strong>26.05</strong></td><td>38.55</td></tr><tr><td>FSDiffusionDet</td><td><strong>57.93</strong></td><td><strong>45.99</strong></td><td><strong>61.33</strong></td><td>53.25</td><td><strong>55.80</strong></td><td><strong>14.66</strong></td><td><strong>54.14</strong></td><td><strong>72.82</strong></td><td><strong>55.80</strong></td><td>15.05</td><td>30.20</td><td><strong>69.64</strong></td><td>24.03</td><td>5.17</td><td>19.23</td><td>38.62</td></tr></tbody></table>

This comparison can be found in Tab. 7.8. This table also includes the Small, Medium, and Large size distinctions from the previous chapters. FSDiffusionDet largely outperforms other methods regarding the object sizes. For small objects, FSDiffusionDet achieves impressive performance on aerial images but lags slightly behind XQSA on natural images. It is particularly noteworthy as it was not designed specifically for small object detection. Another surprising result can be observed on DOTA where medium size objects are better detected than large ones, which is not the case for other datasets. This is unusual compared to all our previous experiments, including attention-based methods. It might result from having too few proposals boxes ($N_p = 300$ in Tab. 7.8 for DOTA), then the model can only focus on small and medium objects as they are more numerous than larger ones. Given the size distribution in DOTA, this is a better compromise as it yields higher overall mAP. From Tab. 7.8, it seems that FSDiffusionDet performs slightly worse than XQSA, DANA and FRW on MS COCO. This is not true as FSDiffusionDet tackles MS COCO as a 20-ways detection problem whereas other approaches only consider an easier 5-ways problem. It is not possible to perform 5-ways episodic evaluation with FSDiffusionDet as all classes must be included during fine-tuning. However, it could be interesting to observe how well attention-based methods perform in the 20-ways settings (it is often challenging to do so with attention-based methods due to memory constraints and long inference time).

Finally, we also provide a qualitative assessment of the performance of FSDiffusionDet on DOTA, DIOR, Pascal VOC and MS COCO in Fig. 7.10. This figure presents novel class detection results on 5 images from each dataset. It is certainly stronger than all previously studied methods (see Fig. 6.13). However, it is not perfect. Some objects are misclassified (see the third row in MS COCO column), some are not detected (DOTA, first row), and there are still false detections (Pascal VOC, 4th row). Nonetheless, these qualitative results are much better than our other approaches and can be considered for actual industrial applications. It strengthens the need for more elaborate fine-tuning strategies and insight into how to design them without too much trial and error. FSDiffusionDet achieves impressive FSOD results, especially on DOTA and DIOR, but this performance was achieved through expensive exploration. It would be of great help to know in advance how a strategy will perform on a given dataset. We started some investigation in this direction with the design of a compatibility
<figure><img src="image_93.png" /><figcaption>Figure 7.10: Qualitative detection results of FSDiffusionDet on DOTA, DIOR, Pascal VOC and MS COCO datasets. The models employed to produce this figure have been fine-tuned with *K* = 10 shots and following the best fine-tuning strategy found in Sec. 7.3.2.4 for each dataset.</figcaption></figure>
score between base and novel classes, taking into account both the overall shift and discrepancies
in the class structures. Lastly, FSDiffusionDet's results are strong enough to tackle more complex
scenarios such as Few-Shot Cross-Domain Adaptation. This will be explored in Sec. 7.4 and should
be continued in future work as well.

7.4 Application to Cross-Domain FSOD

Given the impressive performance of FSDiffusionDet on DOTA and DIOR, it seems tempting to try more difficult setups. Up to now, the methods studied in this project were barely reaching a satisfactory point from an applicative perspective. With FSDiffusionDet, we are past that, and can now consider the Cross-Domain setting. Cross-Domain is especially important for COSE, given the prohibited access to test-time images. The ability to adapt to new domains would be an extremely valuable property for a surveillance system such as CAMELEON. Of course, the domain change would be limited in COSE's applications as the only change between two missions would be the general aspect of the image (i.e., weather, GSD, luminosity, etc.). However, the images will always be aerial taken pointing nadir.

In this section, we tackle the challenging Cross-Domain Few-Shot Object Detection (CD-FSOD) task which is barely untouched in the literature. To this end, we focus on two distinct scenarios, one introduced by [284] with a first training on MS COCO and one specifically designed for COSE's applications where both the source and target domains are aerial datasets. For both scenarios, we first present the dataset used as source and target domains and the experimental setup. Then we provide some experimental results with the FSDiffusionDet baseline. These results are preliminary and promising, further experimentation in this direction is required to better understand this task and further improve FSDiffusionDet in this context. Therefore, we end this section with a summary of the future work that is planned.

7.4.1 MS COCO → Anything

First, we study a general Cross-Domain (CD) setting introduced in the literature by [284]. It consists of training first on MS COCO and then fine-tuning on another dataset with a restricted number of shots. Unlike in the FSOD setting, there is no separation between base and novel classes in CD, all classes of the target domain are considered novel. The benchmark introduced by [284] contains a list of 10 datasets (VisDrone2019 [336], DeepFruits, iWildCam [337], SIXray [109], Fashionpedia [338], Oktoberfest [106], LogoDet-3K [108], CrowdHuman [102], ClipArt [107], KITTI [98]).

7.4.1.1 Cross-Domain Scenarios

The pool of datasets proposed by [284] has a large variety of images, therefore it constitutes a relevant benchmark for CD-FSOD. However, it does not contain any aerial dataset. VisDrone is an aerial image dataset but differs greatly from DOTA or DIOR as its images are taken from a much lower altitude and contain perspective. In addition, 10 datasets make the experiments expensive to run. Thus, we propose a lighter benchmark using VisDrone2019, DeepFruits, SixRay, DOTA, DIOR
<figure><img src="image_94.png" /><figcaption>Figure 7.11: Presentation of the datasets used in our Cross-Domain benchmark: VisDrone2019, DeepFruits, SixRay, and ClipArt.</figcaption></figure>

and ClipArt. We emphasize that dealing with that many datasets is quite challenging as almost every dataset has its own annotation format and data structure. When [284] proposed this benchmark, the authors only provided the list of datasets, without any information about their preparation and split, which makes their experiments very hard to reproduce. On the contrary, we propose for the convenience of future research on CD-FSOD a prepared version of this "meta-dataset"<fnref n="2" /> under the same format (the MS COCO format, which is relatively common in the OD community). In addition to the prepared meta-dataset, we also extend the popular Python package *pycocotools* to help load and explore the datasets.

As mentioned above, we study here 6 cross-domain scenarios with a common base training on MS-COCO. Every scenario has a different target domain represented with one of the following datasets. Fig. 7.11 provides some image examples (without annotations) for VisDrone2019, Deep-Fruits, SixRay, and ClipArt. We refer the reader to Fig. 2.7 for a presentation of DOTA and DIOR. For convenience, we denote these 6 scenarios as COCO → X scenarios.

<footnote n="2">²Link to the Meta-Dataset and Python API package</footnote>
7.4.1.2 Experimental Results

We only experimented with the CD scenario with the baseline of FSDiffusionDet. Specifically, only the detection head is fine-tuned, with 500 proposals. The weights of the model are initialized using the pre-trained weights available on the DiffusionDet repository, except for the last layer of the classification branch which is initialized randomly with the right number of outputs. While this is certainly not optimal for every scenario, this gives a strong baseline to compare with in future experiments.

The results of our experimentation with the cross-domain scenarios can be found in Tab. 7.9 and Fig. 7.12. As for the regular FS experiments, a clear pattern is visible as the number of shots increases. Obviously, the more shots, the better the detection. However, this pattern differs from one scenario to another. For instance, COCO → ClipArt scenario sees very little improvements as the number of shots increases unlike COCO → DOTA and COCO → SIXRay. It is also noteworthy to observe the different behaviors between DOTA and DIOR. Even if DIOR is an "easier" dataset than DOTA (in the sense that higher performance is achieved on DIOR in a regular detection setting), there is a larger difference in the COCO → X cross-domain scenario. Relatively low performance is observed for ClipArt and VisDrone, this is probably due to differences in data preparation compared with [284]. As the author did not provide any information about all the datasets' splits and preparation, we can only guess what they did. For ClipArt, it is slightly different as they leveraged a GAN-augmented version of the dataset which might sensibly boost the detection performance. Finally, for each scenario 5 distinct training were done with varying seeds to check the consistency of our results. Tab. 7.9 gives the average over the 5 runs and a 95% confidence interval. A limited variance between different runs is observed, this means that FSDiffusionDet is not very sensitive to the examples chosen in the support set. This is a crucial property as some few-shot methods depend a lot on the choice of the support set. Of course, most of our experiments should be repeated the same way to strengthen the results, but this quickly becomes expensive in terms of computing resources.

These preliminary results are promising, FSDiffusionDet achieves satisfactory performance with
only a few-annotated examples on various datasets. These datasets are constituted of various kinds
of images, therefore it demonstrates well the adaptation capabilities of FSDiffusionDet. Now for
COSE's application, these results on aerial datasets are particularly encouraging. FSDiffusionDet
achieves impressive performance with only a base training on MS COCO and few examples of either
DOTA or DIOR. Thus, this model could be rapidly fine-tuned for a specific mission by the forces
without declassifying any image. Of course, this is only a baseline and FSDiffusionDet can surely
be improved further. In addition, this scenario starts with a base training on natural images, which
is probably not optimal, instead, we could leverage an aerial dataset as a source model as well.
Incidentally, this will be the subject of the next section.
<table><caption>Table 7.9: Cross-domain performance results on 6 scenarios COCO → DIOR / DOTA / DeepFruits / SIXRay / ClipArt / VisDrone. Results are given for different numbers of shots. Experiments are repeated 5 times for each scenario and shot setting. The average mAP<sub>0.5</sub> is reported with a 95% confidence interval.</caption><thead><tr><th>K Shots</th><th>DIOR</th><th>DOTA</th><th>DeepFruits</th><th>SIXRay</th><th>ClipArt</th><th>VisDrone</th></tr></thead><tbody><tr><td>1</td><td>11.10 ± 0.32</td><td>4.03 ± 0.26</td><td>38.47 ± 1.42</td><td>4.80 ± 0.87</td><td>2.09 ± 0.19</td><td>2.83 ± 0.17</td></tr><tr><td>5</td><td>30.42 ± 0.69</td><td>14.45 ± 0.43</td><td>55.58 ± 1.36</td><td>13.25 ± 1.14</td><td>5.26 ± 0.15</td><td>5.74 ± 0.22</td></tr><tr><td>10</td><td>38.73 ± 0.65</td><td>25.02 ± 0.65</td><td>68.37 ± 2.01</td><td>21.26 ± 1.33</td><td>5.69 ± 0.10</td><td>7.50 ± 0.10</td></tr><tr><td>20</td><td>48.23 ± 0.33</td><td>33.31 ± 0.46</td><td>73.95 ± 0.53</td><td>30.06 ± 1.09</td><td>6.10 ± 0.22</td><td>9.14 ± 0.35</td></tr><tr><td>50</td><td>56.97 ± 0.60</td><td>43.23 ± 0.68</td><td>76.65 ± 0.78</td><td>41.93 ± 1.02</td><td>6.44 ± 0.16</td><td>11.47 ± 0.27</td></tr></tbody></table>

<figure><img src="image_95.png" /><figcaption>Figure 7.12: Cross-domain performance of FSDiffusionDet on multiple scenarios with MS COCO as the source domain. Light areas denote the 95% confidence interval. Concentric circles indicate mAP<sub>0.5</sub> levels.</figcaption></figure>
<table><caption>Table 7.10: FSDiffusionDet Cross-domain results on the scenario DIOR → DOTA. Two settings are compared: with the backbone frozen (left) and the backbone fully fine-tuned (right). Bold values denote the best setting for overall performance on novel classes. Performance is reported with mAP<sub>0.5</sub> values.</caption><thead><tr><th rowspan="3">K shots</th><th colspan="10">DIOR → DOTA</th></tr><tr><th colspan="4">Backbone frozen</th><th colspan="4">Fully fine-tuned</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td>1</td><td>5.41</td><td>2.72</td><td>6.28</td><td>4.51</td><td>5.09</td><td>3.08</td><td>6.72</td><td>4.07</td></tr><tr><td>5</td><td>25.88</td><td>16.99</td><td>31.47</td><td>22.50</td><td>24.90</td><td>15.85</td><td>29.67</td><td>22.27</td></tr><tr><td>10</td><td>31.99</td><td>17.64</td><td>36.90</td><td>31.23</td><td>33.30</td><td>15.97</td><td>37.13</td><td>32.45</td></tr><tr><td>20</td><td>38.77</td><td>21.68</td><td>46.49</td><td>34.79</td><td>41.30</td><td>21.97</td><td>45.90</td><td>41.08</td></tr><tr><td>50</td><td>44.07</td><td>29.22</td><td>52.66</td><td>41.00</td><td>49.22</td><td>29.41</td><td>55.94</td><td>52.82</td></tr></tbody></table>

### 7.4.2 Aerial Cross-Domain

Besides experimenting with COCO → X scenario, we propose another setting specifically designed for aerial images and COSE's application. The idea is to leverage two distinct aerial datasets as source and target domains. In particular, we use DOTA and DIOR to get two scenarios: DOTA → DIOR and DIOR → DOTA. Of course, it would be interesting to leverage other kinds of datasets as well (e.g., xView, VisDrone, etc.), especially as DOTA and DIOR are very similar (mostly overhead urban images and shared classes). Yet, this gives insights into how FSDiffusionDet behaves in fairly simple cross-domain scenarios. We report experiments with these two scenarios in Tabs. 7.10 and 7.11. The results are given for multiple numbers of shots ranging from 1 to 50. In addition, we studied two freezing strategies by fine-tuning only the detection head (i.e., frozen backbone) or the whole model (fully fine-tuned).

The key takeaway from this experiment is that higher performance is achieved in the aerial cross-domain scenarios than with the COCO → X scenario. It seems more profitable to perform base training on a source domain that is similar to the target domain. Of course, compared to the FS performance on DOTA and DIOR, lower quality is achieved in cross-domain scenarios. This is explained first because the images from the two datasets differ, but also because the task is now slightly more complex as all classes of the target dataset are novel. The detection task becomes a 16-way K-shots problem in DIOR → DOTA scenario for instance. In the regular FS setting studied throughout this project, only three classes were selected as novel classes for DOTA, making the classification much easier. Then, from Tabs. 7.10 and 7.11, a contradiction arises, in DIOR → DOTA scenario, the fully fine-tuned model outperforms the model with the backbone frozen, which agrees with the experiments from Sec. 7.3.2.4. However, in the DOTA → DIOR scenario, the inverse is observed. This clearly shows that the freezing sweet spot depends on the source and target domains and cannot be set once and for all. It works in the case of regular few-shot when the source and target domains are identical. So, one could expect the same behavior in cross-domain scenarios with similar source and target domains as it is with DOTA and DIOR. However, in the regular few-shot setting, the number
DOTA → DIOR

<table><caption>Table 7.11: FSDiffusionDet Cross-domain results on the scenario DOTA → DIOR. Two settings are compared: with the backbone frozen (left) and the backbone fully fine-tuned (right). Bold values denote the best setting for overall performance on novel classes. Performance is reported with mAP<sub>0.5</sub> values.</caption><thead><tr><th rowspan="2">K shots</th><th colspan="4">Backbone frozen</th><th colspan="4">Fully fine-tuned</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td>1</td><td><strong>20.18</strong></td><td>5.53</td><td>16.96</td><td>23.43</td><td>9.40</td><td>3.86</td><td>9.15</td><td>8.95</td></tr><tr><td>5</td><td><strong>34.43</strong></td><td>9.99</td><td>31.12</td><td>47.03</td><td>29.57</td><td>8.70</td><td>25.80</td><td>35.76</td></tr><tr><td>10</td><td><strong>41.48</strong></td><td>12.85</td><td>36.62</td><td>53.85</td><td>38.44</td><td>10.50</td><td>32.58</td><td>47.27</td></tr><tr><td>20</td><td><strong>49.00</strong></td><td>16.39</td><td>40.23</td><td>62.79</td><td>45.36</td><td>15.29</td><td>36.51</td><td>55.05</td></tr><tr><td>50</td><td><strong>54.07</strong></td><td>18.70</td><td>43.83</td><td>67.58</td><td>53.51</td><td>19.49</td><td>41.27</td><td>63.04</td></tr></tbody></table>

of classes in the source domain (i.e., the base classes) is always larger than the number of classes in the target domain (i.e., the novel classes). Here, DIOR has more classes than DOTA and this difference may explain the opposite results between the two CD scenarios. Specifically, fine-tuning the model entirely may be beneficial only when the target domain contains fewer classes than the source domain. It could also be caused by different class separations between the datasets. If classes are easily differentiable in DIOR but not in DOTA, it might be difficult to transfer from DIOR to DOTA. These are only conjectures, and they should be taken carefully especially as complex interactions between source and target classes may also cause such behavior. More experiments would be required to analyze and understand this surprising result.

### 7.4.3 Cross-Domain Perspectives

The previous sections have been devoted to cross-domain experiments. These are preliminary but interesting results. They give insight into how difficult this setup is and how fine-tuning strategies can perform. However, plenty of experiments are still necessary. We detail here some of the most relevant perspectives for future CD-FSOD research that we briefly hinted in the previous sections:

1. **Comparison with other FSOD methods:** it would be interesting to compare with other existing FSOD methods, in particular, with attention-based techniques that we studied in depth in Chap. 6. In addition, studying other fine-tuning approaches is required to validate the results found in our experiments.

2. **Transductive inference:** Even though our naive transductive detection did not outperform the fine-tuning strategy in FSOD, it could help in cross-domain scenarios. Indeed, leveraging query images during inference can reduce the discrepancies between source and target domains and improve performance. This has been empirically shown for the classification task, but it remains to be adapted for detection.

3. **Source-target domain compatibility score:** [284] proposes to choose the fine-tuning sweet spot according to the distance between the source and target domains. Specifically, more plasticity is required when domains are farther apart. They compute such distance as the recall of
a pre-trained detection model on MS COCO, applied to the target dataset in a class-agnostic manner. This could be generalized to any source and target domains with a detection model trained on the source domain. However, we would like to emphasize that this should not be called a distance measure between domains as it does not satisfy the symmetry property. Instead, it is a *compatibility measure* as it evaluates how beneficial the source domain is for the adaptation to the target domain. Our cross-domain scenarios on aerial images clearly demonstrate this, as we obtain contradictory conclusions for DOTA → DIOR and DIOR → DOTA scenarios. More plasticity is required for DIOR → DOTA than for DOTA → DIOR, hence, the *compatibility measure* cannot be the same for these two scenarios. In addition, this distance is highly influenced by the detector chosen in the first place, in particular, some models are known to output a lot of duplicate boxes which often boost the recall significantly (see such an analysis in [9]). It would be helpful to come up with a properly defined compatibility measure for a given scenario that does not rely on a detection model and gives coherent hints to obtain an optimal fine-tuning strategy. This measure should also be able to assess the compatibility of the base and novel class set in the regular FSOD setting, as a special case of the cross-domain scenario (source and target domains are identical but the classes change). We are currently working on such a compatibility score based on an overall discrepancy measure between source and target domains and a source-target classes compatibility score.

## 7.5 Conclusion

In this chapter, we have presented thoroughly the basic principle of diffusion models and how they can be leveraged for detection. Then, we proposed a simple fine-tuning strategy to apply Diffusion-Det in the few-shot setting. FSDiffusionDet achieves sensibly higher performance than all previous methods studied in this PhD on aerial images. To understand why, we conducted extensive experimental studies on crucial design choices of our strategy. It highlighted a strong but complex connection between the plasticity of the model and the detection performance. Finally, we applied FSDiffusionDet in several cross-domain scenarios and observed promising results. Again, the plasticity has a great influence on the performance and more experiments must be conducted to understand this relation completely. A possible direction would be to design a compatibility measure between domains and between sets of classes to determine the optimal amount of plasticity required for a given scenario.
# PART III

## RETHINKING INTERSECTION OVER UNION
# SCALE-ADAPTATIVE INTERSECTION OVER UNION

## ABSTRACT

Intersection over Union (IoU) is not an optimal box similarity measure for evaluating and training object detectors. For evaluation, it is too strict with small objects and does not align well with human perception. For training, it provides a poor balance between small and large objects to the detriment of small ones. We propose Scale-adaptive Intersection over Union (SIoU), a parametric alternative that solves the shortcomings of IoU. We provide empirical and theoretical arguments for the superiority of SIoU through in-depth analysis of various criteria.

<figure><img src="image_96.png" /></figure>

P. Le Jeune and A. Mokraoui, "Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime", Submitted at the International Conference on Computer Vision 2023 (ICCV).

<figure><img src="image_97.png" /></figure>

P. Le Jeune and A. Mokraoui, "Extension de l'Intersection over Union pour améliorer la détection d'objets de petite taille en régime d'apprentissage few-shot", GRETSI 2023, XXIXème Colloque Francophone de Traitement du Signal et des Images, Grenoble, France.

## CONTENTS

<table><tr><td>8.1</td><td>Analysis of Intersection over Union</td><td>170</td></tr><tr><td>8.1.1</td><td>Intersection over Union and its Variants</td><td>170</td></tr><tr><td>8.1.2</td><td>Inadequation of IoU for Small Objects and Few-Shot Regime</td><td>171</td></tr><tr><td>8.2</td><td>Scale-Adaptive Intersection over Union</td><td>173</td></tr><tr><td>8.2.1</td><td>Definition of the novel box similarity criterion</td><td>173</td></tr><tr><td>8.2.2</td><td>SIoU Properties</td><td>175</td></tr><tr><td>8.2.3</td><td>Extensions and Generalization of SIoU</td><td>176</td></tr><tr><td>8.3</td><td>Scale-Adaptive Criteria Analysis</td><td>176</td></tr><tr><td>8.3.1</td><td>Response Analysis to Box Shifting</td><td>176</td></tr><tr><td>8.3.2</td><td>Resilience Analysis to Detector Inaccuracy</td><td>177</td></tr><tr><td>8.3.3</td><td>Theoretical study of GIoU</td><td>182</td></tr><tr><td>8.3.4</td><td>Influence of γ and κ on SIoU and GSIoU</td><td>182</td></tr><tr><td>8.4</td><td>SIoU Alignment with Human Perception</td><td>185</td></tr><tr><td>8.4.1</td><td>User Study Presentation</td><td>185</td></tr></table>
<table><tr><td>8.4.2 User Study Insights</td><td>186</td></tr><tr><td>8.5 Experimental Results</td><td>190</td></tr><tr><td>8.5.1 Comparison with Existing Criteria</td><td>192</td></tr><tr><td>8.5.2 Application on Aerial and Natural Images</td><td>192</td></tr><tr><td>8.5.3 γ and κ influence on FSOD Performance</td><td>194</td></tr><tr><td>8.5.4 Additional Experiments with GSIoU Loss</td><td>195</td></tr><tr><td>8.5.5 Evaluation with SIoU</td><td>196</td></tr><tr><td>8.5.6 Discussions and Limitations</td><td>196</td></tr><tr><td>8.6 Conclusion</td><td>198</td></tr></table>

Intersection over Union (IoU) is a box similarity criterion, it measures how well two bounding boxes overlap each other. We already defined it in Chap. 2, but in this chapter we explore its properties thoroughly and highlight some of its flaws when employed as a loss function or as a cornerstone of the evaluation process of detection models. These flaws mainly concern small objects for which IoU is too strict. Therefore, it is particularly relevant to tackle these issues for aerial images and COSE's applications. To address these weaknesses, we propose Scale-Adaptive Intersection over Union (SIoU), a parameterizable criterion that can be set to favor small objects as needed. We start by defining and analyzing the IoU and its variants. Then, we propose our novel criterion SIoU and its properties. Sec. 8.3 presents an original empirical and theoretical study of several box similarity criteria and argues for the superiority of SIoU. Finally, we conduct a user study and experimental analysis to further consolidate the advantages of SIoU over IoU.

## 8.1 Analysis of Intersection over Union

In this section, we first review the definition of IoU and present some of its variants that are available in the literature. Then, we analyze why IoU is not optimal for small objects.

### 8.1.1 Intersection over Union and its Variants

To begin, let us review the definition of existing criteria for box similarity. Originally, the IoU is defined as the intersection area of two sets divided by the area of their union:

$$IoU(A, B) = \frac{|A \cap B|}{|A \cup B|}, \qquad (8.1)$$

where A and B are two sets. Even if there are plenty of applications where IoU is useful (e.g., in statistics where IoU is better known as the Jaccard index), we are mostly interested here in its application in computer vision. In this case, A and B are sets of pixels, and the IoU measures how close they are. When A and B are rectangular boxes, IoU can be computed easily with simple operations on box coordinates (see Eq. (2.7)). This explains why IoU is such a widespread criterion for object detection. It is used as a loss function ($L_{reg} = 1 - IoU$) by several well established detection
frameworks (e.g., [91, 45]). IoU is also involved in the process of example selection during training of most detection methods, i.e., all the ones inspired either by Faster R-CNN [33] or YOLO [34]. In these frameworks, regression loss is computed from the coordinates of proposed boxes and ground truth. Not all pairs of proposals and ground truth are kept for the computation. Only proposals with a sufficient IoU with a ground truth box are selected. Finally, IoU is also used at the heart of the evaluation process. A proposed box is considered a positive detection if it meets two conditions: 1) an IoU greater than a given threshold with a ground truth box, and 2) the same label as this ground truth (see Sec. 2.1.2).

Several attempts were made to improve IoU but existing works mostly focus on the regression loss part, disregarding the other IoU uses in the detection task. First, [92] proposed a generalized version of IoU which yields negative values when boxes do not overlap:

$$GIoU(A, B) = IoU(A, B) - \frac{|C \setminus (A \cup B)|}{|C|}, \qquad (8.2)$$

where C is the convex hull around A and B. This criterion is employed as a loss function by several detection frameworks [45, 37, 339]. It is sometimes also combined with other regression loss as in [340, 59], which both combine it with an L1 regression on box coordinates. Combining IoU loss with other regression terms was also proposed by [93]. They introduce two losses Distance-IoU (DIoU) and Complete-IoU which respectively add an L2 regression term and an aspect ratio penalty to the IoU loss. Recently, α-IoU [94] extends DIoU [93] by proposing a family of losses following the same structure as DIoU with the IoU term raised to the power α:

$$\alpha\text{-IoU}(A, B) = \text{IoU}(A, B)^{\alpha}. \qquad (8.3)$$

Balanced IoU (BIoU) also extends upon DIoU by measuring shifts between the corners of the boxes instead of their centers. Alternatively, Bounded IoU [341] computes an IoU upper bound between a proposal and a ground truth. Other approaches, such as Scale Balanced Loss [342], try to design distance-based loss functions which share properties with IoU, especially its scale-invariance.

All these IoU variants are proposed to improve the regression part of the models. However, IoU is involved in other parts of the framework including example selection, Non-Maximal Suppression, and evaluation. A recent user study [343] indicates that IoU does not completely align with human perception. Humans have strong positional and size preferences based on conceptual information contained in the boxes. It suggests that IoU is not an optimal choice either for example selection or for evaluation as it will lead to detections that do not satisfy human users.

### 8.1.2 Inadequation of IoU for Small Objects and Few-Shot Regime

Object detection is a fundamental task in industry and has applications in many domains such as medical imaging, agriculture, or autonomous driving. However, it is often impracticable or too expensive to build sufficiently large annotated datasets to train detection models. It is therefore
<figure><img src="image_98.png" /><figcaption>Figure 8.1: (Left) Evolution of IoU, NWD [88], the proposed SIoU and α-IoU [94] when a box is shifted from the ground truth box by ε<sub>loc</sub> pixels, for various box width ω ∈ {4, 16, 64, 128} (boxes are squares). (Right) Ratio between pixel localization error ε<sub>loc</sub> and object size ω for a trained detection model on DOTA dataset. Each point represents the localization error of one object in DOTA test set.</figcaption></figure>

crucial to improve data-efficient approaches and particularly Few-Shot Object Detection (FSOD)
methods. However, the limited number of examples provides poor supervision and prevents the
model to learn accurate localization, which is especially problematic for small objects. This difficulty
greatly intensifies in the few-shot regime as shown by Chap. 4. Designing FSOD methods specifically
for the detection of small objects partially solves this issue (see Sec. 6.3.1), but is not enough. One of
the reasons for the poor FSOD performance on small objects is the extensive use of the IoU. Just as
for detection, most FSOD pipelines employ IoU as a regression loss [91, 45]; for example selection
[33, 34, 46]; or as an evaluation criterion, but IoU is not an optimal choice when dealing with small
objects.

IoU has a remarkable property: scale invariance. It means that scaling all coordinates of two bound-
ing boxes by the same amount will not change their IoU. At first glance, this seems a desirable prop-
erty as all objects will be treated identically no matter their size. In practice, it has a fundamental
drawback: small boxes are prone to large IoU changes from only small position or size modifications.
To clarify, let us consider a simple example. Two square boxes of width ω are shifted diagonally by
ε<sub>loc</sub> pixels. In this setup, a 1-pixel shift leads to a larger decrease in IoU when boxes are smaller. This
comes from the scale invariance property, IoU stays constant as the ratio ε<sub>loc</sub> / ω remains fixed. How-
ever, this ratio is not constant for trained detection models, it increases as objects get smaller (see
Fig. 8.1, right), leading to lower IoU values for smaller objects. Hence, small objects are much more
likely to fall under the IoU thresholds which decide if a box is a true or false detection, even though
being satisfactory from a human perspective (see the user study in Sec. 8.4). In addition, Secs. 8.3.1
and 8.3.2 explore the resilience of various criteria to localization inaccuracies and confirm that IoU is not an optimal box similarity measure.

Only a handful of works question the adequation of IoU for object detection. Among those, [92] proposed a generalization of IoU when boxes do not overlap, [88] introduced a novel loss function to target small objects, and [344] proposed a Scale-Sensitive IoU which extended CIoU with an area regulatory factor. In addition, [343] showed that human perception and IoU are not fully aligned. This lack of interest in new criterion design is explained by the great detection performance in the regular setting (i.e., natural images with sufficient annotations). In the few-shot regime, and when targets are small, the flaws of IoU become critical. Therefore, we revisit IoU to improve FSOD methods and focus on aerial images which mostly contain small objects. We propose Scale-adaptive Intersection over Union (SIoU), a novel criterion that can replace IoU for training and evaluating detection models. However, for training we mostly aim at few-shot detection models as small objects are particularly difficult for them. To demonstrate the superiority of the proposed SIoU, Sec. 8.3 compares it with various existing criteria. This section analyzes criteria distributions when exposed to randomly shifted boxes. To our knowledge, this is the first attempt to empirically and theoretically study the distributions of these criteria. The conclusions of this analysis are then compared with human perception through a user study which shows that SIoU aligns better with human appraisal than IoU (see Sec. 8.4). The comparison of these criteria also highlights that SIoU as a loss function can guide training towards small objects better than other criteria and in a more controlled fashion. SIoU loss can be tuned to improve the detection of small objects just as it can be tuned to align with human perception. Finally, these analyses are confirmed by extensive experiments on both aerial images (DOTA and DIOR datasets) and natural images (Pascal VOC and COCO datasets).

## 8.2 Scale-Adaptive Intersection over Union

### 8.2.1 Definition of the novel box similarity criterion

Before introducing the proposed criterion, let us define two bounding boxes $b_1 = [x_1, y_1, w_1, h_1]^T$ and $b_2 = [x_2, y_2, w_2, h_2]^T$ (the prediction box and ground truth respectively), following the box definition from Chap. 2. Similarly, the adjectives small, medium, and large keep the same meaning as in previous chapters: the box $b_i$ is small if $\sqrt{w_ih_i} \le 32$ pixels, medium if $32 < \sqrt{w_ih_i} \le 96$, and large if $\sqrt{w_ih_i} > 96$.

IoU is scale-invariant, hence if $IoU(b_1, b_2) = u$, scaling all coordinates of both boxes by the same factor $k$ will produce the same IoU:

$$IoU(b_1, b_2) = IoU(kb_1, kb_2) = u. \qquad (8.4)$$

However, detection models are not scale-invariant, they do not localize equally well small and large objects. Fig. 8.1 (right) clearly shows that the ratio between the localization error ($\epsilon_{loc} = \|b_1 - b_2\|_1$) and the object size ($\omega = \sqrt{w_2h_2}$) increases as the object becomes smaller. This figure is made with a
detection model trained on DOTA with all annotations. Each point represents the ratio $\frac{\epsilon_{loc}}{\omega}$ for one object in the test set. If the detection model was indeed scale-invariant, the ratio should not change significantly with the object sizes. Hence, because of the scale-invariance property, IoU scores are lower for small objects. It then has several consequences:

1. Bounding boxes output by the model are not considered positive examples during evaluation.

2. Bounding boxes are not selected as positive examples for loss computation, which biases the training towards larger objects.

3. NMS does not filter duplicates of small boxes as their overlap is not high enough.

A way to alleviate these issues is to relax the invariance property of the IoU so it favors more small objects without penalizing large ones. To this end, we propose a novel criterion called Scale-adaptive Intersection over Union (SIoU):

$$ \begin{gathered} \text{SIoU}(b_1, b_2) = \text{IoU}(b_1, b_2)^p \\ \text{with} \quad p = 1 - \gamma \exp\left(-\frac{\sqrt{w_1h_1 + w_2h_2}}{\sqrt{2}\kappa}\right), \end{gathered} \qquad (8.5) $$

where $p$ is a function of the object sizes. Thus, the scores are rescaled according to the object size. $\gamma \in ] -\infty, 1]$ and $\kappa > 0$ are two parameters that control the strength and direction of the rescaling (hence, $p \ge 0$). $\gamma$ governs the scaling for small objects while $\kappa$ controls how fast the behavior of regular IoU is recovered for large objects. Fig. 8.5 (left) in Sec. 8.3.4 shows the evolution of $p$ with object size for various $\gamma$ and $\kappa$. For convenience, we will denote the average object size i.e., the average size of boxes $b_1$ and $b_2$, by $\tau = \frac{w_1h_1+w_2h_2}{2}$.

Of course, there are many valid choices for the exponent $p$. However, we want to ensure some properties for SIoU, which translate into constraints for $p$:

- SIoU should either be higher or lower than IoU when objects are small, but should remain finite, so $p(0) \in \mathbb{R}_+^*$.

- For large objects, SIoU should behave like IoU, $\lim_{\tau \to \infty} p(\tau) = 1$.

- To prevent complete inversion of the order and smooth changes, $p$ should be positive, continuous, and monotonic.

Thus, an exponential response is a natural choice for the design of $p$. Similar forms could be achieved with hyperbolic functions. For instance, $p(\tau) = 1 - \frac{\gamma}{1+\kappa\tau}$ would be a sensible alternative. An inconvenient of these designs is the possibility to only focus on either small or large objects. This is mainly due to the monotonicity of $p$. It can be relaxed to unlock the possibility of targeting objects of a specific size, for instance, with a bell-shaped exponent e.g., $p(\tau) = 1 - \gamma \exp(-\kappa(\tau - \tau_{\text{target}})^2)$. Where $\kappa$ could be understood as a bandwidth parameter around objects of size $\tau_{\text{target}}$. We did not investigate the design of $p$, but experimenting with it would be relevant to better understand the balance between small and large objects during training.
### 8.2.2 SIoU Properties

This new criterion follows the same structure as α-IoU [94], but differs greatly as it sets different powers for different object sizes. SIoU provides a solution for small object detection in the few-shot regime while α-IoU only aims to improve general detection. However, SIoU inherits a few properties from α-IoU.

#### Property 1 (SIoU Relaxation)

Let $b_1$ and $b_2$ be two bounding boxes and introduce $\tau = \frac{w_1h_1+w_2h_2}{2}$ their average area. SIoU preserves the behavior of IoU in certain cases such as:

$$ - \text{IoU}(b_1, b_2) = 0 \Rightarrow \text{SIoU}(b_1, b_2) = \text{IoU}(b_1, b_2) = 0 $$

$$ - \text{IoU}(b_1, b_2) = 1 \Rightarrow \text{SIoU}(b_1, b_2) = \text{IoU}(b_1, b_2) = 1 $$

$$ - \lim_{\tau \to +\infty} \text{SIoU}(b_1, b_2) = \text{IoU}(b_1, b_2) $$

$$ - \lim_{\kappa \to 0} \text{SIoU}(b_1, b_2) = \text{IoU}(b_1, b_2) $$

Property 1 shows that SIoU is sound: it equals IoU when boxes have no intersection and when they perfectly overlap. Therefore, the associated loss function (see Property 2) will take maximal values for boxes that do not overlap and minimum values for identical boxes. In addition, SIoU behaves similarly to IoU when dealing with large objects (i.e., when $\tau \to \infty$). When boxes are large, the power $p$ that rescales the IoU is close to 1. Hence, this change of criterion only impacts small objects. However, when discussing the properties of SIoU, the limit between small/medium/large objects is relative to the choice of $\kappa$. If $\kappa \gg \sqrt{wh}$, even large objects will be rescaled. On the contrary, when $\kappa \to 0$, all objects are treated as large and are not rescaled. In practice, $\kappa$ and $\gamma$ are chosen empirically, but Sec. 8.3 provides useful insights for the choice of these parameters.

#### Property 2 (Loss and gradients reweighting)

Let $\mathcal{L}_{\text{IoU}}(b_1, b_2) = 1 - \text{IoU}(b_1, b_2)$ and $\mathcal{L}_{\text{SIoU}}(b_1, b_2) = 1 - \text{SIoU}(b_1, b_2)$ be the loss functions associated respectively with IoU and SIoU. Let us denote the ratio between SIoU and IoU losses by $W_{\mathcal{L}}(b_1, b_2) = \frac{\mathcal{L}_{\text{SIoU}}(b_1, b_2)}{\mathcal{L}_{\text{IoU}}(b_1, b_2)}$. Similarly, $W_{\nabla}(b_1, b_2) = \frac{|\nabla \mathcal{L}_{\text{SIoU}}(b_1, b_2)|}{|\nabla \mathcal{L}_{\text{IoU}}(b_1, b_2)|}$ denotes the ratio of gradients generated from SIoU and IoU losses:

$$ W_{\mathcal{L}}(b_1, b_2) = \frac{1 - \text{IoU}(b_1, b_2)^p}{1 - \text{IoU}(b_1, b_2)}, \qquad (8.6) $$

$$ W_{\nabla}(b_1, b_2) = p\text{IoU}(b_1, b_2)^{p-1}, \qquad (8.7) $$

$W_{\mathcal{L}}$ and $W_{\nabla}$ are increasing (resp. decreasing) functions of IoU when $p \ge 1$ (resp. $p < 1$) which is satisfied when $\gamma \le 0$ (resp. $\gamma > 0$). As the IoU goes to 1, $W_{\mathcal{L}}$ and $W_{\nabla}$ approaches $p$:

$$ \lim_{\text{IoU}(b_1, b_2) \to 1} W_{\mathcal{L}}(b_1, b_2) = p, \qquad (8.8) $$

$$ \lim_{\text{IoU}(b_1, b_2) \to 1} W_{\nabla}(b_1, b_2) = p. \qquad (8.9) $$

We employ the same tools as in [94] to analyze how SIoU affects the losses and associated gradients. We show in property 2 that their results hold for a non-constant power $p$ as well. From this, it can
be observed that when IoU is close to 1, losses and gradients are both rescaled by $p$. Hence, the gradients coming from objects of different sizes will be rescaled differently. The setting of $\gamma$ and $\kappa$ allows to balance the training towards specific object sizes. Experimental results are provided in Sec. 8.5 to support these findings. Proofs for properties 1 and 2 are available in App. A.

However, order preservingness is not satisfied by using power value changing with the size of the objects. This property ensures that the order given by the IoU is preserved with the novel criterion, e.g., $IoU(b_1, b_2) < IoU(b_1, b_3) \Rightarrow \alpha$-IoU(b_1, b_2) < \alpha$-IoU(b_1, b_3). $\alpha$-IoU preserves the order of IoU, but SIoU does not. We show in App. A that even though this property is not always satisfied, a large proportion of boxes meet the conditions for the order to hold.

### 8.2.3 Extensions and Generalization of SIoU

Finally, SIoU can very well be extended as IoU was with GIoU or DIoU. Note that we only focus on GIoU extension here as DIoU and its variants are composite loss (i.e., sum of multiple loss functions). We provide here an extension following GIoU as it appears especially well-designed for small object detection. When detecting small targets, it is easier for a model to completely miss the object, producing an IoU of 0 no matter how far the predicted box is. On the contrary, GIoU yields negative values for non-intersecting boxes. This produces more relevant guidance during the early phase of training when the model outputs poorly located boxes. Therefore, we extend SIoU by raising GIoU to the same power $p$ as in Eq. (8.5):

$$GSIoU(b_1, b_2) = \begin{cases} GIoU(b_1, b_2)^p & \text{if } GIoU(b_1, b_2) \ge 0 \\ -|GIoU(b_1, b_2)|^p & \text{if } GIoU(b_1, b_2) < 0 \end{cases}. \qquad (8.10)$$

## 8.3 Scale-Adaptive Criteria Analysis

This section analyzes both empirically and theoretically the behaviors of IoU, GIoU [92], $\alpha$-IoU [94], NWD [88], SIoU and GSIoU. We investigate the desirable properties of such criteria for model training and performance evaluation.

### 8.3.1 Response Analysis to Box Shifting

As mentioned in Sec. 8.2, IoU drops dramatically when the localization error increases for small objects. Shifting a box a few pixels off the ground truth can result in a large decrease in IoU, without diminishing the quality of the detection from a human perspective. This is depicted in Fig. 8.1 (left), where plain lines represent the evolution of IoU for various object sizes. These curves are generated by diagonally shifting a box away from the ground truth. Boxes are squares, but similar curves would be observed otherwise. In this plot, boxes have the same size, so when there is no shift in between ($\epsilon_{loc} = 0$), IoU equals 1. However, if the sizes of the boxes differ by a ratio $r$, IoU would peak at $1/r^2$. Other line types represent other criteria. SIoU decreases slower than IoU when $\epsilon_{loc}$ increases
and this is especially true when boxes are small. This holds because $\gamma > 0$, if it was negative, SIoU would adopt the opposite behavior. In addition, the gap between IoU and SIoU is larger when objects are small. Only NWD shares this property, but it only appears when boxes have different sizes (all lines coincide for NWD). Hence, SIoU is the only criterion that allows controlling its decreasing rate, i.e., how much SIoU is lost for a 1-pixel shift. As GIoU and GSIoU values range in [-1, 1], they were not included in Fig. 8.1, but for completeness, they are plotted in Fig. 8.2 along with other criteria.

<figure><img src="image_99.png" /><figcaption>Figure 8.2: Evolution of various criteria (IoU, GIoU, and GSIoU) when a box is shifted from the ground truth box by $\rho$ pixels for various box sizes $\omega \in \{4, 16, 64, 128\}$. With boxes of the same size (left) and different sizes (right).</figcaption></figure>

## 8.3.2 Resilience Analysis to Detector Inaccuracy

Knowing how a criterion responds to shifts and size variations is important to understand what makes a sensible box similarity measure. Pushing beyond the shift analysis, we study empirically and theoretically the criteria's distributions when exposed to detector inaccuracies, i.e., randomly shifted boxes. This setting mimics the inaccuracy of the model either during training or at test time.

### 8.3.2.1 Empirical Protocol

To simplify, let us suppose that all boxes are squares of the same size $\omega$ and can be shifted only horizontally. Similar results are observed by relaxing these constraints, see Sec. 8.3.2.4. A box is then entirely defined by its horizontal position $x$ and its width $\omega$. If a detector is not perfect, it will produce bounding boxes slightly shifted horizontally from the ground truth. To model the detector's inaccuracy, we suppose that the predicted box position is randomly sampled from a Gaussian distribution centered on the ground truth location (which is chosen as 0 without loss of generality):
<figure><img src="image_100.png" /><figcaption>Figure 8.3: Analysis of the distribution of IoU, SIoU, GSIoU and α-IoU when computed on inaccurately positioned boxes. This is done by observing the probability distribution functions for various ω values (**left**), the expectation (**middle**) and standard deviation (**right**) for all criteria. For SIoU and GSIoU, we fixed γ = 0.5 and κ = 64, for α-IoU, α = 3 (as recommended in the original paper [94]). The inaccuracy of the detector is set to σ = 16. Note that the empirical pdfs were smoothed using a Kernel Density Estimator method. This affects particularly IoU, SIoU, and α-IoU for which the actual pdf is defined only on [0, 1]. For the sake of visualization, GIoU and GSIoU were rescaled between 0 and 1 for the expectation and standard deviation plots.</figcaption></figure>

$X \sim N(0, \sigma^2)$ where $\sigma$ controls how inaccurate the model is. We are interested in the distribution of $\mathcal{C} \in \{\text{IoU}, \text{GIoU}, \text{SIoU}, \text{GSIoU}, \alpha\text{-IoU}, \text{NWD}\}$ and how it changes with $\omega$. To this end, let $Z = \mathcal{C}(X)$. More precisely, we are interested in the Probability Density Function (PDF) of $Z$ and its two first moments (which exist because $\mathcal{C}$ is continuous and bounded).

Fig. 8.3 gathers the results of this analysis. It shows the pdf of each criterion for various box sizes (left) along with the evolution of the expectation and standard deviation of $Z$ against $\omega$ (middle and right). Specifically, we randomly sample a large number of boxes and compute the associated criteria values for all $\mathcal{C}$ and boxes. Then, the average and standard deviation are computed to estimate the moment of the criteria' pdfs. This process is repeated for various box sizes $\omega$ to understand how it changes the behaviors of the criteria. From this, it can be noted that the size of the boxes has a large influence on the distributions of all criteria. The expected values of all criteria are monotonically increasing with object size. In particular, small objects have lower expected IoU values than larger ones. This is consistent with the initial assessment from Fig. 8.1 (right) and it validates the choice of $\sigma$ constant for this study (although Sec. 8.3.2 discusses this assumption).

When building detection models, we hope to detect equally well objects of all sizes, this means having a constant expected IoU, no matter the objects' size. This would require the localization error to be an affine function of $\omega$. Of course, the localization error of the detector is likely to depend on $\omega$. However, it cannot be an affine function, otherwise, small objects would be perfectly detected, which is not observed (see Fig. 8.1, right). As SIoU has larger expected values than IoU for small objects, it can compensate for their larger localization errors. The setting of $\gamma$ and $\kappa$ allows
controlling how much small objects are favored (see Fig. 8.5). NWD is not included in these plots as its expected value and variance are constant when dealing with same-size boxes.

### 8.3.2.2 Influence Analysis on the Performance Evaluation

If the expected value of a criterion is too small, it is likely that the boxes will be considered negative detections during evaluation and therefore reduce the performance. Therefore, having a criterion with larger expected values for small objects would better reflect the true performance of a detector. One might think that it would be equivalent to scale-adaptive IoU thresholds during the evaluation, but this is not completely true as the variance of the criteria also differs.

Having an accurate criterion (i.e., with low variance) is crucial for evaluation. Let us take a detector that produces well-localized boxes on average, i.e., on average the criterion computed between the boxes and their corresponding ground truths is above a certain threshold. As the detector is not perfect, it will randomly produce boxes slightly better or slightly worse than the average. If the criterion has a high variance, it will be more likely that poor boxes get scores below the criterion threshold and therefore will be considered negative detections. This will reduce the performance of the detector even though on average, it meets the localization requirements. In addition, a criterion with a higher variance will be less reliable and would produce more inconsistent evaluations of a model. The fact that the IoU variance is high for small objects partly explains why detectors have much lower performance on these objects. Hence, SIoU seems more adapted for evaluation. Of course, using this criterion for evaluation will attribute higher scores for less precise localization of small objects. However, this aligns better with human perception as demonstrated in Sec. 8.4. Employing SIoU in the evaluation process also allows tweaking it for the needs of a specific application.

### 8.3.2.3 Influence Analysis on Training

All criteria discussed above are employed as regression losses in the literature. The loss associated with each criterion $\mathcal{C}$ is $\mathcal{L}_{\mathcal{C}}(b_1, b_2) = 1 - \mathcal{C}(b_1, b_2)$. Therefore, the expected value of the criterion determines the expected value of the loss and thus the magnitude of the gradients. Large values of the criterion give low values for the loss. Now, as the expected values of the criteria change with the object size, the expected values of the losses also change. Small objects generate greater loss values than larger ones on average. However, this is balanced by the fact that fewer small objects are selected as positive examples because the IoU is involved in the selection process. To achieve better detection, training must focus more on small objects. One way to ensure this is to set larger loss values for small objects. Thus, the equilibrium is shifted toward smaller objects and gradients will point to regions where the loss of small objects is lower. As shown in Fig. 8.6 (Sec. 8.3.4), with the right parameters, SIoU can do that. It attributes lower values for small objects while keeping similar values for large ones. The contrast between small and large objects is accentuated and optimization naturally focuses on smaller objects. SIoU's parameters control which object size gets more emphasis. This is closely linked to Property 2 which states that employing SIoU (compared to IoU) reweights the loss and the gradient by $p$. If $\gamma < 0$, $p$ decreases with the size of the objects and
thus the optimization focuses on small objects. This also explains why generalizations of existing
criteria (i.e., with negative values for non-overlapping boxes) often outperform their vanilla version.
Taking IoU and GIoU as examples, the gap between their expected values for small and large objects
is greater with GIoU. It nudges the optimization towards small objects.

8.3.2.4 Inaccuracy Tolerance Assumptions

Several assumptions were made in Sec. 8.3.2 to analyze the criteria for box similarity:

1. Boxes are shifted only horizontally.

2. Boxes have the same size.

3. The detector's inaccuracy is fixed and does not depend on the object size.

The first two assumptions are relatively harmless. Allowing diagonal shifts simply accelerates the IoU drop rate. A 1-pixel diagonal shift is equivalent to a vertical and a horizontal shift. Intuitively, this is similar to a 2-pixels horizontal shift. However, this is not true because, with a 1-pixel diagonal shift, the area of intersection decreases slower than with a 2-pixels horizontal shift. Following the notations from Sec. 8.3.2, the intersection between two boxes of width $\omega$ diagonally shifted by $\rho$ pixels is $(\omega - \rho)^2 = \omega^2 - 2\omega\rho + \rho^2$ while the intersection between same boxes horizontally shifted by $2\rho$ pixels is $\omega(\omega - 2\rho) = \omega^2 - 2\omega\rho$. To ensure that this does not question the conclusions of Sec. 8.3.2.1, Fig. 8.4a compares the expected values and variances of IoU and GIoU with horizontal and diagonal shifts. Similar behaviors are observed with and without diagonal shifting. The only difference is that the expected values of the criteria for diagonally shifted boxes are lower as the shifts get larger. It also increases the variances as the distributions are more spread. Then, relaxing the second constraint results in slightly different distributions, but with similar behavior. Having boxes of different sizes only changes the maximum value of the criteria. If boxes have different sizes, the maximum value must be smaller than 1. Therefore, the expected values approach smaller values than 1 as objects get larger. The variance is reduced as the range of criteria values is smaller (see Fig. 8.4b).

It is less straightforward that the conclusions hold without the last assumption. In the analysis from Sec. 8.3.2.1, we assume that the inaccuracy of the detector is fixed. This means that the detector generates randomly shifted boxes by the same number of pixels on average no matter the size of the object. This is certainly false, in practice, in terms of absolute distance, detectors are better with smaller objects. However, the inaccuracy cannot simply be proportional to object sizes because small objects would then be perfectly detected. Thus, we tried to change the inaccuracy of the detector as an affine function of the box width: $\sigma(\omega) = \sigma_0 + \lambda\omega$. We choose to set $\sigma_0$ to the fixed value of $\sigma$ used in Sec. 8.3.2.1 and $\lambda = 1/4$. This setting reflects better the inaccuracy of a true detector. The expected values and standard deviations of IoU, SIoU, GIoU, and GSIoU with this inaccuracy setting are plotted in Fig. 8.4c. The main difference with fixed inaccuracy is that expected values do not approach 1 as object size gets larger, instead they tend towards lower values. It also leads to non-zero variance for large objects. However, for small objects, the curves of the different criteria are mostly unchanged, and the conclusions formulated in Sec. 8.3.2.1 are still valid.
<figure><img src="image_101.png" /><figcaption>(a) IoU and GIoU expected values and standard deviation with horizontally and diagonally shifted boxes.</figcaption></figure>

<figure><img src="image_102.png" /><figcaption>(b) IoU and GIoU expected values and standard deviation with and without boxes of the same size.</figcaption></figure>

<figure><img src="image_103.png" /><figcaption>(c) IoU, SIoU, GIoU, and GSIoU expected values and standard deviation with the detector's inaccuracy modeled as an affine function, $\sigma(\omega) = \sigma_0 + \lambda\omega$ ($\sigma_0 = 16, \lambda = \frac{1}{4}$).</figcaption></figure>

Figure 8.4: Relaxing the constraints for criteria' distribution analysis.
### 8.3.3 Theoretical study of GIoU

In the previous section, we derived the statistics of several box similarity criteria from empirical simulations. However, the criteria probability distribution functions and first moments can also be derived theoretically. We provide such results for GIoU in Proposition 1.

**Proposition 1 (GIoU's distribution)**

Let $b_1 = (0, y_1, w_1, h_1)$ be a bounding box horizontally centered and $b_2 = (X, y_2, w_2, h_2)$ another bounding box randomly positioned, with $X \sim \mathcal{N}(0, \sigma^2)$ and $\sigma \in \mathbb{R}_+^*$. Let's suppose that the boxes are identical squares, shifted only horizontally (i.e., $w_1 = w_2 = h_1 = h_2$ and $y_1 = y_2$). Let $Z = \mathcal{C}(X)$, where $\mathcal{C}$ is the generalized intersection over union. The probability density function of $Z$ is given by:

$$d_Z(z) = \frac{2\omega}{(1+z)^2\sqrt{2\pi}\sigma} \exp\left(-\frac{1}{2}\left[\frac{\omega(1-z)}{\sigma(1+z)}\right]^2\right). \qquad (8.11)$$

The two first moments of $Z$ exist and are given by:

$$\mathbb{E}[Z] = \frac{2}{\pi^{3/2}} G_{3,2}^{2,3} \begin{pmatrix} 2a^2 & \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & 0 & 0 \end{pmatrix}, \qquad (8.12)$$

$$\mathbb{E}[Z^2] = 1 - \frac{8a}{\sqrt{2\pi}} + \frac{16a^2}{\pi^{3/2}} G_{3,2}^{2,3} \begin{pmatrix} 2a^2 & -\frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & 0 & -\frac{1}{2} \end{pmatrix}, \qquad (8.13)$$

where $G$ is the Meijer G-function [345] (see its definition in Eq. (B.4), App. B) and $a = \frac{\sigma}{\omega}$.

The proof of this proposition and derivations for other criteria are available in App. B. The theoretical expressions completely agree with empirical results, which confirms the soundness of our simulations.

Other criteria do not have closed forms for their first and second moments. Nonetheless, we provide in Tab. 8.1 their expressions keeping the integrals as simple as possible, which allows relatively easy evaluation. In addition, we provide the expression of the pdf for each criterion. The setup remains identical as in Proposition 1, the boxes are only horizontally shifted and have the same width $\omega$. For clarity, we also give simple expressions of each criterion in such a setup (see Tab. 8.1).

### 8.3.4 Influence of $\gamma$ and $\kappa$ on SIoU and GSIoU

In the previous discussion, SIoU and GSIoU are parametrized with $\gamma = 0.5$ and $\kappa = 64$; however, these two parameters have an influence on the distribution and moments of SIoU and GSIoU. First, following the analysis from Sec. 8.3.2.1, Fig. 8.5 investigates the influence of $\gamma$ and $\kappa$ on SIoU behavior. Fig. 8.5a shows the value of $p$, the expectation and variance of SIoU against object size for $\gamma \in \{1.0, 0.75, 0.5, 0.1, 0.0, -0.25, -2, -4\}$. $p$ is a function of the average area of the boxes ($\tau = \frac{w_1h_1+w_2h_2}{2}$) and for simplicity we suppose here that the boxes are squares of the same width $\omega$, hence $\sqrt{\tau} = \omega$. Then $p$ can be viewed as a function of $\omega$: $p(\omega) = 1 - \gamma \exp(-\omega/\kappa)$. For negative values of $\gamma$, $p$ decreases from $p(0) = 1 - \gamma$ to 1, small objects get higher exponents in comparison with larger objects. On the contrary, when $\gamma > 0$, $p$ increases from $p(0) = 1 - \gamma$ to 1. Changing $\gamma$
<table><caption>Table 8.1: Criteria expression, probability distribution and first two moments for IoU, GloU, SloU, and GSIoU. These are valid for the comparison of same size square boxes of width $\omega$ randomly shifted horizontally. Random shifts are sampled from a centered Gaussian distribution of variance $\sigma^2$ and $a = \sigma/\omega$. $F_X$ and $d_X$ are respectively the cumulative and probability density function of X.</caption><thead><tr><th>Criterion Expression</th><th>Probability Density Function</th><th>E[Z]</th><th>E[Z<sup>2</sup>]</th></tr></thead><tbody><tr><td>IoU</td><td><b>C: R → [0, 1]</b><br>x ↦ max (<sup>ω − |x|</sup>⁄<sub>ω + |x|</sub>)<br><br><b>C: R → [−1, 1]</b><br>x ↦ max (<sup>ω − |x|</sup>⁄<sub>ω + |x|</sub>)</td><td>d<sub>Z</sub>(z) = 2 [ (1 − F<sub>X</sub>(ω<sup>1 − z</sup>⁄<sub>1 + z</sub>)δ<sub>0</sub>(z) + 1/R<sub>+</sub>(z) (1 + z)<sup>2</sup> d<sub>X</sub>(ω<sup>1 − z</sup>⁄<sub>1 + z</sub>)) ]<br><br>d<sub>Z</sub>(z) = <sup>4ω</sup>⁄<sub>(1 + z)<sup>2</sup>√2σ</sub> exp ( − <sup>1</sup>⁄<sub>2</sub> [ ω(1 − z) ]<sup>2</sup> σ(1 + z) ]<sup>2</sup>)</td><td>E[Z] = <sup>4</sup>⁄<sub>√2πa</sub> ∫<sub>0</sub><sup>1</sup> <sup>1</sup>⁄<sub>1 + u</sub> e<sup>−u<sup>2</sup>⁄<sub>2a<sup>2</sup></sub></sup> du − erf(<sup>1</sup>⁄<sub>√2a</sub>)<br><br>E[Z<sup>2</sup>] = erf(<sup>1</sup>⁄<sub>√2a</sub>) − <sup>8a</sup>⁄<sub>√2πa</sub> [ <sup>1</sup>⁄<sub>2</sub> e<sup>−<sup>1</sup>⁄<sub>2a<sup>2</sup></sub></sup> − 2 ∫<sub>0</sub><sup>1</sup> <sup>1</sup>⁄<sub>(1 + u)<sup>3</sup></sub> e<sup>−<sup>u<sup>2</sup></sup>⁄<sub>2a<sup>2</sup></sub></sup> du ]<br><br>E[Z<sup>2</sup>] = 1 − <sup>8a</sup>⁄<sub>√2π</sub> + <sup>16a<sup>2</sup></sup>⁄<sub>π<sup>3/2</sup></sub> G<sub>3,2</sub><sup>2,3</sup> ( 2a<sup>2</sup> |<sup>1</sup>⁄<sub>2</sub> 0 |<sup>1</sup>⁄<sub>2</sub> − <sup>1</sup>⁄<sub>2</sub> )</td></tr><tr><td>GloU</td><td><b>C: R → [0, 1]</b><br>x ↦ { <sup>(ω − |x|)</sup>⁄<sub>(ω + |x|)</sub> <sup>p</sup> if ω − |x| ≥ 0 otherwise</td><td>d<sub>Z</sub>(z) = 2 [ (1 − F<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>)δ<sub>0</sub>(z) + 1/R<sub>+</sub>(z) (1 + z<sup>1/p</sup>)<sup>2</sup> d<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>)]</td><td>E[Z] = 2ω ∫<sub>0</sub><sup>1</sup> (<sup>1 − u</sup>⁄<sub>1 + u</sub>)<sup>p</sup> e<sup>−<sup>u<sup>2</sup></sup>⁄<sub>2a<sup>2</sup></sub></sup> du<br><br>E[Z] = 2ω ∫<sub>0</sub><sup>1</sup> (<sup>1 − u</sup>⁄<sub>1 + u</sub>)<sup>p</sup> e<sup>−<sup>u<sup>2</sup></sup>⁄<sub>2a<sup>2</sup></sub></sup> du<br><br>E[Z<sup>2</sup>] = 2ω ∫<sub>0</sub><sup>1</sup> (<sup>1 − u</sup>⁄<sub>1 + u</sub>)<sup>2p</sup> e<sup>−<sup>u<sup>2</sup></sup>⁄<sub>2a<sup>2</sup></sub></sup> du</td></tr><tr><td>SIoU</td><td><b>C: R → [−1, 1]</b><br>x ↦ { <sup>(ω + |x|)</sup>⁄<sub>0</sub> <sup>p</sup> if ω − |x| ≥ 0 otherwise</td><td>d<sub>Z</sub>(z) = 2 [ (1 − F<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>)δ<sub>0</sub>(z) + 1/R<sub>+</sub>(z) (1 + z<sup>1/p</sup>)<sup>2</sup> d<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>) ]<br><br>d<sub>Z</sub>(z) = 2 [ (1 − F<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>)δ<sub>0</sub>(z) + 1/R<sub>+</sub>(z) (1 + z<sup>1/p</sup>)<sup>2</sup> d<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>) ]<br><br>d<sub>Z</sub>(z) = 2 [ (1 − F<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>)δ<sub>0</sub>(z) + 1/R<sub>+</sub>(z) (1 + z<sup>1/p</sup>)<sup>2</sup> d<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>) ]<br><br>d<sub>Z</sub>(z) = 2 [ (1 − F<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>)δ<sub>0</sub>(z) + 1/R<sub>+</sub>(z) (1 + z<sup>1/p</sup>)<sup>2</sup> d<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>) ]<br><br>d<sub>Z</sub>(z) = 2 [ (1 − F<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>)δ<sub>0</sub>(z) + 1/R<sub>+</sub>(z) (1 + z<sup>1/p</sup>)<sup>2</sup> d<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>) ]<br><br>d<sub>Z</sub>(z) = 2 [ (1 − F<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>)δ<sub>0</sub>(z) + 1/R<sub>+</sub>(z) (1 + z<sup>1/p</sup>)<sup>2</sup> d<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>) ]<br><br>d<sub>Z</sub>(z) = 2 [ (1 − F<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>)δ<sub>0</sub>(z) + 1/R<sub>+</sub>(z) (1 + z<sup>1/p</sup>)<sup>2</sup> d<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>) ]<br><br>d<sub>Z</sub>(z) = 2 [ (1 − F<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>)δ<sub>0</sub>(z) + 1/R<sub>+</sub>(z) (1 + z<sup>1/p</sup>)<sup>2</sup> d<sub>X</sub>(ω<sup>1 − z<sup>1/p</sup></sup>⁄<sub>1 + z<sup>1/p</sup></sub>) ]</td><td>E[Z] = 2ω [ ∫<sub>0</sub><sup>1</sup> (<sup>1 − u</sup>⁄<sub>1 + u</sub>)<sup>p</sup> e<sup>−<sup>u<sup>2</sup></sup>⁄<sub>2a<sup>2</sup></sub></sup> du − ∫<sub>1</sub><sup>+∞</sup> (<sup>1 − u</sup>⁄<sub>1 + u</sub>)<sup>p</sup> e<sup>−<sup>u<sup>2</sup></sup>⁄<sub>2a<sup>2</sup></sub></sup> du ]<br><br>E[Z<sup>2</sup>] = 2ω [ ∫<sub>0</sub><sup>1</sup> (<sup>1 − u</sup>⁄<sub>1 + u</sub>)<sup>2p</sup> e<sup>−<sup>u<sup>2</sup></sup>⁄<sub>2a<sup>2</sup></sub></sup> du − ∫<sub>1</sub><sup>+∞</sup> (<sup>1 − u</sup>⁄<sub>1 + u</sub>)<sup>2p</sup> e<sup>−<sup>u<sup>2</sup></sup>⁄<sub>2a<sup>2</sup></sub></sup> du ]</td></tr></tbody></table>
<figure><img src="image_104.png" /><figcaption>Figure 8.5: Influence of $\gamma$ and $\kappa$ on the expected value and standard deviation of SIoU.</figcaption></figure>

also influences the distribution of SIoU. As $\gamma$ increases, the expected value for small objects increases as well, while the variance decreases.

Fig. 8.5b shows the same curves for $\kappa \in \{8, 16, 32, 64, 256\}$. $\kappa$ controls how fast $p$ approaches 1 and therefore, changing $\kappa$ simply shifts the curves of expectation and variance accordingly. As $\kappa$ increases, IoU's behavior is retrieved for larger objects reducing the expected value of SIoU. The variance is not changed much by $\kappa$, but it slightly shifts the maximum of the curve, i.e., the object size for which SIoU's variance is maximum.

Fig. 8.6 also provides pdfs plots for various object sizes for SIoU and GSIoU, in addition to expectation and variance comparison between existing criteria. For this figure, $\gamma = -3$ and $\kappa = 16$. This figure echoes Fig. 8.3 which plots the same curves but with $\gamma = 0.5$ and $\kappa = 64$.
<figure><img src="image_105.png" /><figcaption>Figure 8.6: Analysis of the distribution of IoU, SIoU, GSIoU and α-IoU when computed on inaccurately positioned boxes. This is done by observing the probability distribution functions (pdfs) for various ω values (**left**), the expectation (**middle**) and standard deviation (**right**) for all criteria. For SIoU and GSIoU, we fixed γ = -4 and κ = 16, for α-IoU, α = 3 (as recommended in the original paper [94]). The inaccuracy of the detector is set to σ = 16. Note that the empirical pdfs were smoothed using a Kernel Density Estimator method. This affects particularly IoU, SIoU and α-IoU as the actual PDF is defined only on [0, 1]. For the sake of visualization, GIoU and GSIoU were rescaled between 0 and 1 for the expectation and standard deviation plots.</figcaption></figure>

**8.4 SIoU Alignment with Human Perception**

As discussed in Sec. 8.3.2.1, having an accurate criterion i.e., one with low variance, is crucial for
evaluation. However, such a criterion must also align with human perception. Most image process-
ing models are destined to assist human users. Thus, to maximize the usefulness of such models, the
evaluation process should align as closely as possible with human perception. To assess the agree-
ment between the criteria and human perception, we conducted a user study in which participants
had to rate on a 1 to 5 scale (i.e., from *very poor* to *very good*) how a bounding box localizes an object.
Specifically, an object is designated by a green ground truth box and a red box is randomly sampled
around the object (i.e., with random IoU with the ground truth). Then, the participants rate how well
the red box localizes the object within the green one. The study gathered 75 different participants
and more than 3000 individual answers. We present here the main conclusion of this study.

**8.4.1 User Study Presentation**

**8.4.1.1 Experimental Protocol**

To carry out the user study about detection preferences, we developed a Web App<fnref n="1" /> to gather partic-
ipants' answers. Each participant had to sign in with a brief form. They are asked about their age
and whether they are familiar with image analysis. This information is only meant to detect rating
differences between different population groups (see Fig. 8.11, last two rows). After completing the
form, participants are brought to the rating page (see Fig. 8.8). On this page, one image is visible

<footnote n="1">¹The web app is available [here](#).</footnote>
with two bounding boxes drawn on it. A green one, which represents the ground truth annotation of an object, and a red one randomly shifted and deformed. Each participant must rate how well the red box is detecting the object inside the green box. The rating is done on a 5-levels scale, going from *very poor* to *very good*. A set of 50 different images is shown to each participant. After 25 images, the experiment changes slightly: the background image is replaced by a completely black image. This would remove any contextual bias coming from the variety of objects inside the green box. We refer to the two phases of the experiment as respectively, the phases with and without context. The red boxes are sampled around the green box, but to enforce a uniform distribution of the IoU with the green box, a random IoU value *u* is first uniformly sampled between 0 and 1. Then, we randomly generate a red box that has an IoU *u* with the green box (direct box sampling does not produce uniformly distributed IoU values). Participants are instructed to answer quickly and are provided with examples for each possible rating (see Fig. 8.8). The images and the annotations are randomly picked from the DOTA dataset.

### 8.4.1.2 General Statistics about Participants

The study gathered 75 participants and a total of 3136 individual answers (because some participants did not complete the entire experiment). The age of the participants ranges from 21 to 64 years old with an average of 31. Approximately half (37) of the participants are versed in computer vision or image analysis, we will refer to this group as the *expert group*. On average, the response time is 10.3s per evaluated image during the first phase of the experiment (when a background image is visible). It drops to 7.2s when the image is replaced with a uniform background during the second phase. This time difference suggests that humans do take into account the contextual information of the image inside their decision-making process, which agrees with the findings of [343].

## 8.4.2 User Study Insights

Human perception does not fully align with IoU. People tend to be more lenient than IoU towards small objects. Specifically, comparing a small and a large box with the same IoU with respect to their own ground truth, people will rate the small one better. This suggests that IoU is too strict for small objects in comparison with human perception. From a human perspective, precise localization seems less important for small objects. Fig. 8.9 represents the relative gap of IoU (left) and SIoU (right) values for each object size and rating. The relative differences $c_{s,r}$ are computed against the average IoU (or SIoU) value per rating:

$$c_{s,r} = \frac{\mathcal{C}_{s,r} - \sum_{s} \mathcal{C}_{s,r}}{\sum_{s} \mathcal{C}_{s,r}}, \qquad (8.14)$$

where $\mathcal{C}_{s,r}$ is the average criterion value ($\mathcal{C} \in \{\text{IoU}, \text{SIoU}\}$) for objects of size *s* and rating *r*. IoU values for small objects (in orange) are lower than for large objects (in red) for all rating *r*. For a human to give a rating *r* to a box, it requires that a box overlaps less with the ground truth (according to IoU) if the boxes are small. SIoU compensates for this trend (see Fig. 8.9).
IoU User Study

<figure><img src="image_106.png" /></figure>

Image 8 / 50

Rate the quality of the **red rectangle (algorithm)** with respect to the **green rectangle (expert)** on the proposed scale. Examples and protocole are available below.

<figure><img src="image_107.png" /></figure>

Acceptable

If none of the options seems valid, you can skip the image by clicking twice on the same grade. In that case, you must
explain what was wrong with that image before clicking "Next image".

<figure><img src="image_108.png" /><figcaption>Figure 8.7: Rating page of the Web App with an example of an image and two rectangular bounding boxes. The user is asked to rate the quality of the red box compared to the green one on a 5-levels scale going from *very poor* to *very good*.</figcaption></figure>

<figure><img src="image_109.png" /><figcaption>Figure 8.8: Examples given to the participants of the user study. The IoU between the green and red boxes are 0.1, 0.25, 0.5, 0.75, and 0.9 for the ratings from very poor to very good respectively.</figcaption></figure>
<table><caption>Table 8.2: Kendall's τ correlation between various criteria and human rating *r*. For SIoU, $γ = 0.2$ and $κ = 64$, for α-IoU, $α = 3$.</caption><thead><tr><th></th><th><em>r</em></th><th>IoU</th><th>SIoU</th><th>α-IoU</th><th>NWD</th></tr></thead><tbody><tr><th><em>r</em></th><td>1.000</td><td>0.674</td><td><strong>0.701</strong></td><td>0.674</td><td>0.550</td></tr><tr><th><strong>IoU</strong></th><td>0.674</td><td>1.000</td><td>0.892</td><td>0.997</td><td>0.474</td></tr><tr><th><strong>SIoU</strong></th><td>0.701</td><td>0.892</td><td>1.000</td><td>0.892</td><td>0.576</td></tr><tr><th><strong>α-IoU</strong></th><td>0.674</td><td>0.997</td><td>0.892</td><td>1.000</td><td>0.472</td></tr><tr><th><strong>NWD</strong></th><td>0.550</td><td>0.474</td><td>0.576</td><td>0.472</td><td>1.000</td></tr></tbody></table>

However, according to SIoU, the same overlapping value triggers the same human rating no matter the object sizes (see Fig. 8.9). Or at least the required criterion value gap between small and large objects is reduced. We can observe that it is more difficult to compensate the gaps for higher rating values (especially with *r* = 4). This means that human appraisal is not identically in favor of small objects. Instead, it seems more and more in favor of the small objects until the boxes are nearly identical (rating *r* = 5). It would be relevant to extend SIoU further to take this into account and achieve even better alignment with human perception. Anyway, SIoU is much better aligned than IoU with the human rating as for a rating *r*, the difference of SIoU for small and large objects is below 5% while with IoU, it is always above 15%. This means that with IoU, if we have two predicted boxes for one ground truth, the IoUs of the predicted boxes with the ground truth can vary from about 15% without changing the human rating. Such a difference in SIoU would result in different human ratings for the two predicted boxes. This phenomenon is much more problematic with α-IoU and NWD. This makes them poor choices for the evaluation process as they are poorly aligned with human perception.

Similar charts are available in Fig. 8.10, only with SIoU but with various values of $γ$. In the previous paragraph, we set $γ = 0.2$. Choosing higher $γ$ values would reverse the trend and produce a criterion even more lenient than humans for small objects. It will also decrease further SIoU's variance. However, this setting has been chosen to maximize the alignment with human perception. SIoU with $γ = 0.2$ correlates better with human rating compared with other criteria. As the rating is an ordered categorical variable, we use the Kendall rank correlation to make the comparison. The correlation between the human rating *r* and each criterion can be found in Tab. 8.2. SIoU with $γ = 0.2$ and $κ = 64$ aligns best with human perception and has a low variance (see Sec. 8.3.4). This showcases the superiority of SIoU over existing criteria. It should be preferred over IoU to assess the performance of models on all visual tasks that employ IoU within their evaluation process. It supports recent findings that show misalignment between IoU and human preference [343].

### 8.4.2.1 Factor Analysis

To validate our previous experiments, we investigate the potential influence of external factors on human ratings which could bias the results. Specifically, we are interested here in several variables: the object size, the presence of contextual information, the expertise, and the age of the participants.
<figure><img src="image_110.png" /><figcaption>Figure 8.9: Criteria' scores for different object sizes and human ratings $r \in \{1, 2, 3, 4, 5\}$ (top). Relative gap with the criterion value averaged over the object sizes ($c_r = 1/3(c_{S,r} + c_{M,r} + c_{L,r})$) (bottom).</figcaption></figure>

<figure><img src="image_111.png" /><figcaption>Figure 8.10: Relative gap with the SIoU values averaged over the object sizes, for various $\gamma$ values.</figcaption></figure>

Of course, the object size seems to have an influence on the human rating (according to the previous section), but the idea is to show it quantitatively. To this end, Tab. 8.3 gathers the average rating $r$ under different groupings (by object size, presence of contextual information, the expertise of the participants, and age of the participants). In addition, the average value for each criterion is given for each group. IoU value is close to 0.5 for every group as expected (boxes were chosen to have a uniform IoU distribution). However, values of other criteria vary from one group to another. This is especially true for scale-dependent criteria (SIoU and NWD) on different object size groups. To check whether the different groups are statistically different, we conducted one-way ANOVA tests on the four variables from Tab. 8.3. The results confirm that the mean ratings for various object sizes are statistically different ($p < 8.4 \times 10^{-26}$<fnref n="2" />). The tests find no statistical differences for the participant expertise ($p < 0.47$), and the presence of contextual information ($p < 0.28$). However, there is a significant difference between age groups ($p < 0.02$), but its influence on the ratings is

<footnote n="2">²p stands here for the p-value of the statistical test here, not the exponent from SIoU's definition.</footnote>
<table><caption>Table 8.3: Average rating and criteria values for different groupings of the variables of interest (object size, presence of contextual information, expertise and age of the participants).</caption><thead><tr><th></th><th></th><th>r</th><th>IoU</th><th>SIoU</th><th>NWD</th><th>α-IoU</th></tr></thead><tbody><tr><td rowspan="3">Object size</td><td>Small</td><td>3.406</td><td>0.507</td><td>0.550</td><td>0.610</td><td>0.203</td></tr><tr><td>Medium</td><td>3.158</td><td>0.502</td><td>0.532</td><td>0.424</td><td>0.199</td></tr><tr><td>Large</td><td>2.824</td><td>0.491</td><td>0.500</td><td>0.151</td><td>0.189</td></tr><tr><td rowspan="2">Context</td><td>w/o context</td><td>3.144</td><td>0.504</td><td>0.531</td><td>0.397</td><td>0.197</td></tr><tr><td>w/ context</td><td>3.112</td><td>0.496</td><td>0.523</td><td>0.390</td><td>0.197</td></tr><tr><td rowspan="2">Expertise</td><td>Inexperienced</td><td>3.104</td><td>0.493</td><td>0.520</td><td>0.392</td><td>0.194</td></tr><tr><td>Expert</td><td>3.152</td><td>0.507</td><td>0.535</td><td>0.395</td><td>0.200</td></tr><tr><td rowspan="3">Age</td><td>(10, 25]</td><td>3.215</td><td>0.504</td><td>0.531</td><td>0.397</td><td>0.196</td></tr><tr><td>(25, 40]</td><td>3.078</td><td>0.496</td><td>0.524</td><td>0.390</td><td>0.198</td></tr><tr><td>(40, 65]</td><td>3.085</td><td>0.501</td><td>0.529</td><td>0.394</td><td>0.197</td></tr></tbody></table>

limited compared to object sizes. This confirms values inside Tab. 8.3 as the older age groups tend to give lower ratings. Our goal here is not to infer anything about the reasons for this difference, but this fact should be kept in mind before drawing any conclusion. In addition, this suggests the need of distinct alignments given what population group is the end user of a system. Having a parameterizable evaluation process (e.g., with SIoU) could help design models that better satisfy their users.

To visualize better the alignment of the various criteria with the human perception, Fig. 8.11 plots the rating values against the criteria value. For clarity, random vertical shifts are added to rating values to distinguish between the values of each variable and data points. From this figure, it is clear that the IoU is not a perfect criterion as a wide range of IoU values is attributed to the same rating value. It seems also clear that contextual information and participant expertise do not introduce much change in the human rating. However, age does have a small influence on the rating, but this is more blatant with the object size: the average IoU value for a rating *r* decreases with the object size (this is visible with the black vertical lines in Fig. 8.9). This completely agrees with the statistical test results. SIoU compensates for this trend and produces more aligned averages for the different object sizes. NWD has the same effect but largely reverses the trend in the other direction.

## 8.5 Experimental Results

To support our analysis from Sec. 8.3, we conduct various experiments, mainly on aerial images with DOTA and DIOR datasets. To showcase the versatility of SIoU, we also experiment with natural images on Pascal VOC and COCO datasets. We would like to emphasize that the goal of SIoU, as a loss, is primarily to improve the performance of FSOD and not regular object detection. While SIoU is beneficial for any detection task for evaluation purposes, it is designed to address the extreme challenge of detecting small objects in the few-shot regime. Therefore, most of our experiments focus on the few-shot setting. However, we also report results in regular object detection to display the
<figure><img src="image_112.png" /><figcaption>Figure 8.11: Rating against IoU, SIoU ($\gamma = 0.2, \kappa = 64$), NWD and $\alpha$-IoU ($\alpha = 3$) values, overall and for different groupings of the variables of interest (object size, presence of contextual information, expertise and age of the participants). Colors represent different values for each variable. A legend for each row is included in the right-most column of the figure. For the *Age* variable, the participants have been separated into three groups of the same size.</figcaption></figure>
<table><caption>Table 8.4: Average criteria (IoU, SIoU, NWD and α-IoU) values for different object sizes and ratings.</caption><thead><tr><th></th><th>r</th><th>IoU</th><th>SIoU</th><th>NWD</th><th>α-IoU</th></tr></thead><tbody><tr><td rowspan="5">Small Objects</td><td>1</td><td>0.214</td><td>0.262</td><td>0.346</td><td>0.013</td></tr><tr><td>2</td><td>0.279</td><td>0.330</td><td>0.415</td><td>0.035</td></tr><tr><td>3</td><td>0.449</td><td>0.498</td><td>0.551</td><td>0.108</td></tr><tr><td>4</td><td>0.584</td><td>0.627</td><td>0.683</td><td>0.234</td></tr><tr><td>5</td><td>0.746</td><td>0.776</td><td>0.822</td><td>0.435</td></tr><tr><td rowspan="5">Medium Objects</td><td>1</td><td>0.223</td><td>0.257</td><td>0.160</td><td>0.016</td></tr><tr><td>2</td><td>0.299</td><td>0.335</td><td>0.230</td><td>0.039</td></tr><tr><td>3</td><td>0.474</td><td>0.506</td><td>0.361</td><td>0.127</td></tr><tr><td>4</td><td>0.651</td><td>0.677</td><td>0.575</td><td>0.306</td></tr><tr><td>5</td><td>0.771</td><td>0.791</td><td>0.716</td><td>0.470</td></tr><tr><td rowspan="5">Large Objects</td><td>1</td><td>0.245</td><td>0.258</td><td>0.015</td><td>0.025</td></tr><tr><td>2</td><td>0.322</td><td>0.334</td><td>0.036</td><td>0.051</td></tr><tr><td>3</td><td>0.517</td><td>0.527</td><td>0.121</td><td>0.168</td></tr><tr><td>4</td><td>0.713</td><td>0.720</td><td>0.318</td><td>0.383</td></tr><tr><td>5</td><td>0.766</td><td>0.772</td><td>0.422</td><td>0.480</td></tr></tbody></table>

potential of SIoU. For the few-shot experiments, we choose Cross-Scale Query Support Alignment (XQSA) (see Sec. 6.3) as a baseline, but a comparison with the other attention methods from Chap. 6 is available in Sec. 8.5.4.1.

### 8.5.1 Comparison with Existing Criteria

To begin, we compare the few-shot performance on DOTA with various loss functions based on the criteria discussed in Sec. 8.3. The result of these experiments is available in Tab. 8.5. The criteria are divided into two groups, generalized (i.e., which is not 0 when boxes do not overlap; it therefore includes NWD) and vanilla criteria. As discussed in Sec. 8.3.2.1, the generalized versions of the criteria outperform their original counterparts and therefore should be compared separately. Scale-adaptive criteria (SIoU and GSIoU) largely outperform other losses on novel classes and especially on small objects. For SIoU and GSIoU, we choose $γ = -3$ and $κ = 16$ according to a series of experiments conducted on DOTA to determine their optimal values (see Sec. 8.3.4). It is important to point out the relatively good performance of NWD despite not checking all the desirable properties highlighted in Sec. 8.3.

### 8.5.2 Application on Aerial and Natural Images

As the previous set of experiments was only carried out on DOTA, we showcase the versatility of GSIoU on three other datasets: DIOR, Pascal VOC and COCO. As it is clear that generalized criteria achieve higher performance, the comparison here is only done between GIoU and GSIoU. The results of this comparison are available in Tab. 8.6.

The large improvements found for DOTA translate to DIOR as well, especially for small objects. For Pascal VOC and COCO, similar gains are observed for small objects, but the improvement is
<table><caption>Table 8.5: Few-shot performance comparison between several criteria: IoU, α-IoU, SIoU, NWD, GIoU, and GSIoU trained on DOTA. mAP is reported with a 0.5 IoU threshold for small (S), medium (M), large (L), and all objects.</caption><thead><tr><th rowspan="2">Loss</th><th rowspan="2">All</th><th colspan="3">Base classes</th><th colspan="3">Novel Classes</th></tr><tr><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td><strong>IoU</strong></td><td>50.67</td><td><strong>25.83</strong></td><td>57.49</td><td>68.24</td><td>32.41</td><td>10.06</td><td>47.87</td><td>67.09</td></tr><tr><td><strong>α-IoU</strong></td><td>46.72</td><td>13.24</td><td>55.21</td><td><strong>69.94</strong></td><td>33.95</td><td>12.58</td><td>46.58</td><td><strong>74.50</strong></td></tr><tr><td><strong>SIoU</strong></td><td><strong>53.62</strong></td><td>24.07</td><td><strong>61.91</strong></td><td>67.34</td><td><strong>39.05</strong></td><td><strong>16.59</strong></td><td><strong>54.42</strong></td><td><strong>74.49</strong></td></tr><tr><td><strong>NWD</strong></td><td>50.79</td><td>19.19</td><td>58.90</td><td>67.90</td><td>41.65</td><td>28.26</td><td>50.16</td><td>65.06</td></tr><tr><td><strong>GIoU</strong></td><td>52.41</td><td><strong>26.94</strong></td><td>61.17</td><td>63.00</td><td>41.03</td><td>24.01</td><td><strong>52.13</strong></td><td>69.78</td></tr><tr><td><strong>GSIoU</strong></td><td><strong>52.91</strong></td><td>22.14</td><td><strong>61.19</strong></td><td><strong>66.02</strong></td><td><strong>45.88</strong></td><td><strong>34.83</strong></td><td>51.26</td><td><strong>70.78</strong></td></tr></tbody></table>

<table><caption>Table 8.6: Few-shot performance on four datasets: DOTA, DIOR, Pascal VOC and COCO. GIoU and GSIoU losses are compared. mAP is reported with a 0.5 IoU threshold and for all object sizes.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">XQSA</th><th colspan="4">Base classes</th><th colspan="3">Novel Classes</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td rowspan="2">DOTA</td><td>w/ GIoU</td><td>52.41</td><td><strong>26.94</strong></td><td>61.17</td><td>63.00</td><td>41.03</td><td>24.01</td><td><strong>52.13</strong></td><td>69.78</td></tr><tr><td>w/ GSIoU</td><td><strong>52.91</strong></td><td>22.14</td><td><strong>61.19</strong></td><td><strong>66.02</strong></td><td><strong>45.88</strong></td><td><strong>34.83</strong></td><td>51.26</td><td><strong>70.78</strong></td></tr><tr><td rowspan="2">DIOR</td><td>w/ GIoU</td><td>58.90</td><td>10.38</td><td>40.76</td><td>80.44</td><td>47.93</td><td>9.85</td><td>47.61</td><td>68.40</td></tr><tr><td>w/ GSIoU</td><td><strong>60.29</strong></td><td><strong>11.28</strong></td><td><strong>43.24</strong></td><td><strong>81.63</strong></td><td><strong>52.85</strong></td><td><strong>13.78</strong></td><td><strong>53.73</strong></td><td><strong>71.22</strong></td></tr><tr><td rowspan="2">Pascal</td><td>w/ GIoU</td><td>51.09</td><td><strong>13.93</strong></td><td><strong>40.26</strong></td><td>62.01</td><td>48.42</td><td>18.44</td><td>36.06</td><td>59.99</td></tr><tr><td>w/ GSIoU</td><td><strong>54.47</strong></td><td>13.88</td><td>40.13</td><td><strong>66.82</strong></td><td><strong>55.16</strong></td><td><strong>22.94</strong></td><td><strong>36.24</strong></td><td><strong>67.40</strong></td></tr><tr><td rowspan="2">COCO</td><td>w/ GIoU</td><td>19.15</td><td><strong>8.72</strong></td><td>22.50</td><td>30.59</td><td>26.25</td><td>11.96</td><td>23.95</td><td>38.60</td></tr><tr><td>w/ GSIoU</td><td><strong>19.57</strong></td><td>8.41</td><td><strong>23.02</strong></td><td><strong>31.07</strong></td><td><strong>27.11</strong></td><td><strong>12.81</strong></td><td><strong>26.02</strong></td><td><strong>39.20</strong></td></tr></tbody></table>

limited overall (i.e., disregarding the object size) as natural datasets contain larger objects. It is worth mentioning that these improvements on Pascal VOC and COCO require a different tuning of SIoU. $γ = -3$ and $κ = 16$ produce mitigated results with these datasets, and $γ = -1$ and $κ = 64$ is a more sensible choice. This was predictable as the objects in Pascal VOC and COCO are substantially larger than in DOTA and DIOR. This can also explain the slightly smaller gains on DIOR compared to DOTA. Finding optimal values of $γ$ and $κ$ could yield slightly better performance on DIOR. The right balance depends on the proportion of small, medium and large objects in the datasets. With natural images which contain fewer small objects, the training balance does not need to be shifted as much as for aerial images.

Obviously, one may see this as a constraint: SIoU introduces two novel hyper-parameters and their values change depending on which dataset is used. However, the tuning of SIoU is straightforward, as lower values of $γ$ and $κ$ skew the training towards smaller objects, and in practice few experiments are enough to find near-optimal values. Given the impressive gains obtained on small objects in the few-shot regime, it is certainly worth it. Clearly, it would be even better to extend SIoU to be parameter-free or to have a pre-defined way of setting $γ$ and $κ$, for instance, based on the object size distribution inside a dataset.
### 8.5.3 $\gamma$ and $\kappa$ influence on FSOD Performance

As discussed above, the setting of $\gamma$ and $\kappa$ is crucial for training. Therefore, we conducted various experiments on DOTA to find the best parameters for GSIoU loss. We intentionally include extreme values of $\gamma$ and $\kappa$ to demonstrate the behavior of SIoU. The results can be found in Tabs. 8.7 and 8.8. This shows that the optimal values for DOTA dataset are $\gamma = -3$ and $\kappa = 8$. However, $\kappa = 16$ is also a good choice and is more consistent across datasets. Thus, we choose to keep $\gamma = -3$ and $\kappa = 16$ for other experiments. An exhaustive grid search should be done to find even better settings. Our search was sparse and a better combination of $\gamma$ and $\kappa$ probably exists, yet our sub-optimal setup already yields significant improvement for small object detection in the FS regime.

<table><caption>Table 8.7: Evolution of the few-shot performance (XQSA with GSIoU loss) on DOTA for various values of $\gamma$ ($\kappa = 16$ is fixed). mAP is reported with a 0.5 IoU threshold and for all object sizes.</caption><thead><tr><th rowspan="2">γ</th><th colspan="4">Base classes</th><th colspan="4">Novel Classes</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td><b>0.5</b></td><td>47.09</td><td>21.29</td><td>54.67</td><td>65.48</td><td>30.50</td><td>8.83</td><td>44.97</td><td>65.89</td></tr><tr><td><b>0.25</b></td><td>45.94</td><td>21.60</td><td>54.39</td><td>63.40</td><td>30.96</td><td>12.53</td><td>42.37</td><td>64.14</td></tr><tr><td><b>0</b></td><td>52.41</td><td>26.94</td><td>61.17</td><td>63.00</td><td>41.03</td><td>24.01</td><td>52.13</td><td>69.78</td></tr><tr><td><b>-0.5</b></td><td>52.80</td><td><b>27.16</b></td><td>61.19</td><td>64.61</td><td>41.06</td><td>25.20</td><td>50.18</td><td><b>72.04</b></td></tr><tr><td><b>-1</b></td><td>53.03</td><td>23.20</td><td>61.53</td><td>66.68</td><td>42.77</td><td>27.55</td><td><b>52.01</b></td><td>70.76</td></tr><tr><td><b>-2</b></td><td><b>54.06</b></td><td>23.68</td><td><b>62.69</b></td><td>66.62</td><td>43.67</td><td>30.04</td><td>51.69</td><td>69.66</td></tr><tr><td><b>-3</b></td><td>52.91</td><td>22.14</td><td>61.19</td><td>66.02</td><td><b>45.88</b></td><td><b>34.83</b></td><td>51.26</td><td>70.78</td></tr><tr><td><b>-4</b></td><td>53.59</td><td>22.50</td><td>62.48</td><td>66.18</td><td>42.43</td><td>27.56</td><td>51.79</td><td>68.70</td></tr><tr><td><b>-9</b></td><td>53.11</td><td>20.98</td><td>62.13</td><td><b>67.00</b></td><td>42.63</td><td>30.53</td><td>48.89</td><td>68.62</td></tr></tbody></table>

<table><caption>Table 8.8: Evolution of the few-shot performance (XQSA with GSIoU loss) for various values of $\kappa$ ($\gamma = -2$ is fixed).</caption><thead><tr><th rowspan="2">κ</th><th colspan="4">Base classes</th><th colspan="4">Novel Classes</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td><b>4</b></td><td>51.65</td><td>21.50</td><td>59.76</td><td>65.85</td><td>42.98</td><td>30.33</td><td>48.57</td><td><b>73.41</b></td></tr><tr><td><b>8</b></td><td>52.70</td><td>21.96</td><td>61.49</td><td>66.43</td><td><b>44.16</b></td><td><b>31.35</b></td><td>50.70</td><td>71.99</td></tr><tr><td><b>16</b></td><td><b>54.06</b></td><td><b>23.68</b></td><td>62.69</td><td>66.62</td><td>43.67</td><td>30.04</td><td>51.69</td><td>69.66</td></tr><tr><td><b>32</b></td><td>53.88</td><td>22.33</td><td><b>63.00</b></td><td><b>67.35</b></td><td>37.36</td><td>23.65</td><td>44.60</td><td>66.29</td></tr><tr><td><b>64</b></td><td>52.82</td><td>21.79</td><td>61.46</td><td>66.77</td><td>43.68</td><td>29.43</td><td><b>52.47</b></td><td>69.46</td></tr><tr><td><b>128</b></td><td>53.42</td><td>21.73</td><td>62.90</td><td>66.75</td><td>41.32</td><td>26.85</td><td>49.40</td><td>70.38</td></tr></tbody></table>

#### Influence of $\gamma$ and $\kappa$ on FSOD performance on Pascal VOC dataset

The search conducted above was carried out on DOTA dataset. It transposes nicely on DIOR dataset as well. Yet, these two datasets are similar. They both contain aerial images with some classes in common, but most importantly, the size of their objects are highly similar. When applied to different datasets, these results may not hold. For instance, Pascal VOC requires other combinations of $\gamma$ and $\kappa$ to outperform the training with GIoU. This is shown in Tab. 8.9.
<table><caption>Table 8.9: Few-shot performance on Pascal VOC dataset with different values of γ and κ.</caption><thead><tr><th rowspan="2">Loss function</th><th colspan="4">Base Classes</th><th colspan="4">Novel Classes</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td>GIoU</td><td>51.09</td><td>13.93</td><td>40.26</td><td>62.01</td><td>48.42</td><td>18.44</td><td>36.06</td><td>59.99</td></tr><tr><td>GSIoU γ = -3, κ = 16</td><td>45.22</td><td>10.06</td><td>34.85</td><td>57.10</td><td>43.16</td><td>14.89</td><td>33.92</td><td>54.16</td></tr><tr><td>GSIoU γ = -1, κ = 64</td><td>54.47</td><td>13.88</td><td>40.13</td><td>66.82</td><td>55.16</td><td>22.94</td><td>36.24</td><td>67.40</td></tr><tr><td>GSIoU γ = 0.5, κ = 64</td><td>56.97</td><td>13.88</td><td>40.75</td><td>70.31</td><td>55.36</td><td>20.25</td><td>36.85</td><td>68.05</td></tr></tbody></table>

## 8.5.4 Additional Experiments with GSIoU Loss

### 8.5.4.1 Changing the Few-Shot Approach

To support the versatility of GSIoU, we also experiment with several few-shot approaches. We selected three FSOD techniques that we implemented within the AAF framework: Feature Reweighting [220] (FRW), Dual-Awareness Attention [234] (DANA) and our Cross-scale Query-Support Alignment (XQSA). We train all of them with GIoU and GSIoU as regression losses and provide the results in Tab. 8.10. Except for DANA, GSIoU sensibly improves detection performance, especially for small objects. The results with DANA are surprising, and it would be of great interest to investigate the reasons behind this below-par performance.

<table><caption>Table 8.10: Performance comparison with three different FSOD methods: Feature Reweighting [220] (FRW), Dual Awareness Attention [234] (DANA) and our Cross-scale Query-Support Alignment (XQSA), trained with GIoU and GSIoU. mAP is reported with a 0.5 IoU threshold for small (S), medium (M), large (L) and all objects.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">XQSA</th><th rowspan="2">All</th><th colspan="4">Base classes</th><th colspan="4">Novel Classes</th></tr><tr><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td rowspan="2">FRW</td><td>w/ GIoU</td><td>34.60</td><td>16.15</td><td>48.61</td><td>59.00</td><td>32.00</td><td>15.29</td><td>44.50</td><td>54.77</td></tr><tr><td>w/ GSIoU</td><td>30.36</td><td>11.94</td><td>44.30</td><td>54.87</td><td>32.94</td><td>16.69</td><td>42.87</td><td>62.64</td></tr><tr><td rowspan="2">DANA</td><td>w/ GIoU</td><td>48.09</td><td>27.34</td><td>66.06</td><td>68.00</td><td>44.49</td><td>30.10</td><td>52.24</td><td>74.40</td></tr><tr><td>w/ GSIoU</td><td>50.10</td><td>32.19</td><td>65.46</td><td>67.77</td><td>41.40</td><td>21.07</td><td>54.80</td><td>75.23</td></tr><tr><td rowspan="2">XQSA</td><td>w/ GIoU</td><td>45.30</td><td>26.94</td><td>61.17</td><td>63.00</td><td>41.03</td><td>24.01</td><td>52.13</td><td>69.78</td></tr><tr><td>w/ GSIoU</td><td>43.42</td><td>22.14</td><td>61.19</td><td>66.02</td><td>45.88</td><td>34.83</td><td>51.26</td><td>70.78</td></tr></tbody></table>

### 8.5.4.2 Regular Object Detection on DOTA and DIOR

GSIoU is not only beneficial for FSOD, but it also improves the performance of regular object detection methods. Tab. 8.11 compares the performance of FCOS [45] trained on DOTA and DIOR with GIoU and GSIoU. The same pattern is visible as we get better performance with GSIoU. However, the gain for small objects is not as large as for FSOD. Nevertheless, it suggests that other tasks relying on IoU could also benefit from GSIoU.

Obviously, further experiments are required to showcase the superiority of SIoU/GSIoU in regular data settings, especially with other detection frameworks and datasets. We did investigate with YOLO and DiffusionDet, but we only achieved mitigated results, even sometimes in favor of GIoU.
<table><caption>Table 8.11: Regular Object Detection performance on DOTA and DIOR datasets with GIoU and GSIoU ($\gamma = -3$ and $\kappa = 16$) losses. mAP is computed with several IoU thresholds (0.5 to 0.95) as it is commonly done in regular detection.</caption><thead><tr><th rowspan="2">FCOS</th><th colspan="4">DOTA</th><th colspan="4">DIOR</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td>w/ GIoU</td><td>34.9</td><td>17.4</td><td>36.6</td><td>43.3</td><td>48.1</td><td>10.1</td><td>40.3</td><td>63.2</td></tr><tr><td>w/ GSIoU</td><td><strong>36.8</strong></td><td><strong>17.5</strong></td><td><strong>40.4</strong></td><td><strong>45.2</strong></td><td><strong>49.2</strong></td><td><strong>11.0</strong></td><td><strong>41.2</strong></td><td><strong>66.1</strong></td></tr></tbody></table>

One possibility for this is the use of IoU in the example selection process. In most detection frameworks, only the best predicted bounding boxes, according to the IoU, are selected to compute the loss (this is detailed in Sec. 2.1.3.5 and especially Tab. 2.2). As mentioned previously, it would be relevant to study the impact of SIoU on this process as well. However, FCOS, on which the AAF framework is based, does not rely on IoU for the example selection. Instead, it selects as positive examples all predicted boxes whose center falls into a ground truth box. Hence, an IoU-based example selector probably hinders the benefits of SIoU while the FCOS's selection is likely a better match. This should be investigated in depth in future work.

### 8.5.5 Evaluation with SIoU

In this section, we present some of the results reported in previous section using SIoU as the evaluation criterion. Specifically, instead of choosing an IoU threshold to decide if a box is a positive or negative detection, an SIoU threshold is employed. For the sake of comparison, we kept the same thresholds as in Tabs. 8.5, 8.6 and 8.11, i.e., 0.5 for Few-Shot methods and 0.5:0.95 for regular object detection. The results are available in Tabs. 8.12 to 8.14. The conclusions from Sec. 8.5 still hold, and the superiority of GSIoU over other criteria is clear. However, a few changes are noticeable. First, SIoU loss seems to perform better than IoU. This is expected since the model is directly optimized to satisfy this criterion. Then, when evaluated with SIoU, models trained with NWD perform well. Indeed, NWD puts a lot of emphasis on size matching during training, and less on position. Therefore, it is logical to observe better performance compared to other losses when using SIoU as the evaluation criterion.

One crucial point is that SIoU evaluation mostly changes the score for small objects. SIoU behaves like IoU for large objects, therefore relatively small changes are visible for medium and large objects. Overall, the scores are higher than with IoU as the expected value of SIoU is higher than IoU. The important point to note is that the gap between small and large objects performance is reduced and aligns better with human perception.

### 8.5.6 Discussions and Limitations

As mentioned in Sec. 8.3.2.1 SIoU is a better choice for performance analysis. However, as IoU is almost the only choice in literature for evaluation, we must use it as well for a fair comparison with existing works. Nonetheless, Tabs. 8.12 to 8.14 provide results from Tabs. 8.5, 8.6 and 8.11 using
<table><caption>Table 8.12: Few-shot performance comparison between several criteria: IoU, α-IoU, SIoU, NWD, GIoU and GSIoU trained on DOTA. mAP is reported with a 0.5 SIoU threshold for small (S), medium (M), large (L), and all objects.</caption><thead><tr><th rowspan="2">Loss</th><th rowspan="2">All</th><th colspan="3">Base classes</th><th colspan="3">Novel Classes</th></tr><tr><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td><strong>IoU</strong></td><td>55.81</td><td>35.03</td><td>62.57</td><td>70.05</td><td>39.10</td><td>18.58</td><td>53.93</td><td>68.83</td></tr><tr><td><strong>α-IoU</strong></td><td>53.05</td><td>20.60</td><td>61.05</td><td><strong>72.41</strong></td><td>41.93</td><td>20.99</td><td>55.74</td><td>76.79</td></tr><tr><td><strong>SIoU</strong></td><td><strong>59.77</strong></td><td>36.38</td><td>67.29</td><td>70.06</td><td>49.51</td><td>31.06</td><td>62.53</td><td><strong>77.24</strong></td></tr><tr><td><strong>NWD</strong></td><td>58.80</td><td>34.16</td><td>66.81</td><td>70.05</td><td>53.66</td><td>42.02</td><td>62.53</td><td>68.92</td></tr><tr><td><strong>GIoU</strong></td><td>59.27</td><td><strong>44.07</strong></td><td><strong>66.91</strong></td><td>65.46</td><td>49.02</td><td>35.10</td><td>57.58</td><td>74.30</td></tr><tr><td><strong>GSIoU</strong></td><td>59.32</td><td>35.32</td><td>66.29</td><td>69.03</td><td><strong>57.70</strong></td><td><strong>46.77</strong></td><td><strong>65.56</strong></td><td>73.67</td></tr></tbody></table>

<table><caption>Table 8.13: Regular Object Detection performance on DOTA and DIOR datasets with GIoU and GSIoU (γ = -3 and κ = 16) losses. mAP is computed with several SIoU thresholds (0.5 to 0.95) as it is commonly done in regular detection.</caption><thead><tr><th rowspan="2">FCOS</th><th rowspan="2">All</th><th colspan="3">DOTA</th><th colspan="3">DIOR</th></tr><tr><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td>w/ GIoU</td><td>43.9</td><td>27.4</td><td>46.5</td><td>47.2</td><td>54.5</td><td>17.6</td><td>49.8</td><td>66.4</td></tr><tr><td>w/ GSIoU</td><td><strong>45.4</strong></td><td>27.7</td><td><strong>50.2</strong></td><td><strong>49.2</strong></td><td><strong>55.4</strong></td><td><strong>18.0</strong></td><td><strong>50.1</strong></td><td><strong>69.2</strong></td></tr></tbody></table>

<table><caption>Table 8.14: Few-shot performance on three datasets: DOTA, DIOR, Pascal VOC and COCO. GIoU and GSIoU losses are compared. mAP is reported with a 0.5 SIoU threshold and for various object sizes.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">XQSA</th><th rowspan="2">All</th><th colspan="3">Base classes</th><th colspan="3">Novel Classes</th></tr><tr><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td rowspan="2">DOTA</td><td>w/ GIoU</td><td>59.27</td><td><strong>44.07</strong></td><td>66.91</td><td>65.46</td><td>49.02</td><td>35.10</td><td>57.58</td><td><strong>74.30</strong></td></tr><tr><td>w/ GSIoU</td><td><strong>59.32</strong></td><td>35.32</td><td><strong>66.29</strong></td><td><strong>69.03</strong></td><td><strong>57.70</strong></td><td><strong>46.77</strong></td><td><strong>65.56</strong></td><td>73.67</td></tr><tr><td rowspan="2">DIOR</td><td>w/ GIoU</td><td>62.06</td><td>17.49</td><td>45.55</td><td>82.22</td><td>53.81</td><td>23.79</td><td>53.46</td><td>71.63</td></tr><tr><td>w/ GSIoU</td><td><strong>63.81</strong></td><td><strong>17.77</strong></td><td><strong>49.62</strong></td><td><strong>82.53</strong></td><td><strong>58.79</strong></td><td><strong>25.60</strong></td><td><strong>59.28</strong></td><td><strong>73.78</strong></td></tr><tr><td rowspan="2">Pascal</td><td>w/ GIoU</td><td>55.51</td><td>26.10</td><td><strong>46.82</strong></td><td>64.31</td><td>52.43</td><td>28.97</td><td>40.73</td><td>62.58</td></tr><tr><td>w/ GSIoU</td><td><strong>58.74</strong></td><td><strong>27.47</strong></td><td>46.56</td><td><strong>68.93</strong></td><td><strong>58.92</strong></td><td><strong>31.36</strong></td><td><strong>41.65</strong></td><td><strong>69.71</strong></td></tr><tr><td rowspan="2">COCO</td><td>w/ GIoU</td><td>21.46</td><td>12.77</td><td>24.79</td><td>31.86</td><td>29.21</td><td>17.36</td><td>27.62</td><td>40.05</td></tr><tr><td>w/ GSIoU</td><td><strong>21.97</strong></td><td><strong>12.80</strong></td><td><strong>25.72</strong></td><td><strong>32.35</strong></td><td><strong>29.94</strong></td><td><strong>18.87</strong></td><td><strong>29.93</strong></td><td><strong>40.47</strong></td></tr></tbody></table>
SIoU as the evaluation criterion in Sec. 8.5.5. They agree with the IoU evaluation and strengthen the conclusions of our experiments. While these results are promising, we must emphasize a few limitations of SIoU and our study. First, SIoU requires a slight tuning to get the best performance, even if that tuning is quite straightforward and mostly depends on the size distribution in the target images. SIoU allows being more lenient with small objects for evaluation ($\gamma \ge 0$), and stricter for training ($\gamma \le 0$) to prioritize the detection of small targets. However, this is a small price to pay compared to the performance gains obtained on aerial datasets and especially on small objects. Second, another limitation is its application to regular object detection. While this works relatively well with FCOS, it does not show consistent results with other frameworks. It could likely be explained as, in these frameworks, IoU plays a crucial role in the example selection and loss computation. More investigation is required to answer this question. Similarly, Non-Maximal Suppression leveraged IoU as well, and therefore, an implementation of NMS with SIoU instead could also help greatly for the detection task, especially for small objects. Finally, we discussed the alignment of SIoU with human perception for evaluation, i.e., inside the computation of the mAP. However, recent works [8, 9] question the soundness of this metric and propose alternatives that are not necessarily based on IoU. It would be relevant to study them as well and understand how well they align with human perception in order to design more user-oriented detection models.

## 8.6 Conclusion

In this chapter, we highlighted the weaknesses of Intersection over Union both for training and evaluating few-shot object detection models. As an alternative, we proposed Scale-adaptive Intersection over Union (SIoU), a criterion that changes with the object size. We performed an in-depth empirical and theoretical study of several criteria and showed that SIoU has desirable properties for model evaluation that other criteria have not. This is confirmed by a user study that shows a better alignment of SIoU with human appreciation. In addition, we experimented thoroughly with SIoU as a loss function and obtained impressive performance gains on small object detection in the few-shot regime. This is particularly helpful for applications on aerial images, especially as it is compatible with the attention mechanism that we presented in Chap. 6 to improve small object detection in the few-shot regime.
# PART IV

## PROTOTYPING AND INDUSTRIAL APPLICATION
CHAPTER 9

# INTEGRATION IN COSE PROTOTYPES

## ABSTRACT

Detection models are often heavy and are not well suited for COSE's application. In this chapter, we first present in detail the CAMELEON system and its constraints. Then, we study the influence of the model size on the performance and present useful tools and tricks to accelerate the inference. Finally, we explain how the detection models are deployed inside the CAMELEON prototype and how they perform on aerial images.

## CONTENTS

<table><tr><td>9.1 CAMELEON Aerial Intelligence System.</td><td>202</td></tr><tr><td>9.1.1 Presentation of the system</td><td>202</td></tr><tr><td>9.1.2 CAMELEON Image Specifications and Constraints</td><td>204</td></tr><tr><td>9.2 Reducing Object Detection Model Size for Edge Computing</td><td>205</td></tr><tr><td>9.2.1 Object Detection Accuracy/Speed Tradeoff</td><td>205</td></tr><tr><td>9.2.2 Knowledge Distillation</td><td>208</td></tr><tr><td>9.3 Inference Acceleration with TensorRT</td><td>209</td></tr><tr><td>9.4 Object Detection Pipeline for CAMELEON Prototype</td><td>212</td></tr><tr><td>9.4.1 Deployment with TensorRT</td><td>212</td></tr><tr><td>9.4.2 Detection Performance Evaluation and Profiling</td><td>215</td></tr><tr><td>9.5 Conclusion</td><td>218</td></tr></table>

Up to this point, the contributions of this project have been mostly research-oriented and a significant amount of work is still required to apply the developed techniques on real-case scenarios. Therefore, in this chapter, we present the engineering part of this project, which focuses on applying object detection algorithms inside the CAMELEON system. In particular, we detail the architecture of the CAMELEON system and the constraints associated. In light of these constraints, we can adapt
object detection algorithms for COSE's applications. This mainly involves reducing the size of the models and their inference speed while preserving their detection quality. Of course, this is no easy task, but fortunately, there exist tools to help with this process. We present these tools and their principle briefly, before explaining how they can be leveraged for deploying object detection models on edge devices within the CAMELEON system. Finally, we provide some guidelines for future improvements of optimized detection models and how to deploy few-shot models as well.

# 9.1 CAMELEON Aerial Intelligence System

## 9.1.1 Presentation of the system

CAMELEON is a high-resolution aerial camera system that aims to be embedded on various types of carriers such as tactical helicopters (e.g., Airbus H215), patrol or intelligence aircraft (e.g., Airbus A400M or ATL-2), and tactical drones (e.g., Safran's Patroller). Its main objectives are to provide precise 2D and 3D models of geographical areas at needs. Often, satellite imagery is insufficient as it has a low ground resolution, can be outdated, or simply unavailable (e.g., due to weather conditions). In such situations, airborne reconnaissance systems are vital. In practice, these kinds of systems have military applications (theater cartography, tactical intelligence, etc.) but also civil and commercial (maritime surveillance, search and rescue, fire monitoring, etc.). Such aerial surveillance systems already exist (e.g., GlobalScanner product by COSE as well). However, their specifications are often limited in light of the recent progress in sensor resolution and quality. As an example, optronics systems often provide extreme ground resolution but are limited to a small area. CAMELEON aims at improving the compromise between the swath and ground resolution. Specifically, CAMELEON will embed up to 6 camera sensors (each producing ~ 100M pixels images). This allows for an extra wide field of view and significant overlapping between images (which is crucial for 3D modeling). This involves dealing with huge amounts of data, which requires carefully designed hardware and software. The CAMELEON system is constituted of two major components, the on-board components and the ground component. Both will be presented in the following sections.

The on-board segment of CAMELEON is the heart of the system, it includes a sensor block, a dedicated computer and a user interface:

- **Sensor Block**: this unit gathers all the sensors of the system along with the mechanical parts and motors used for controlling their orientations. All the sensors are attached at the core of a 3-axis gimbal suspension. These three axes are controlled by motors to compensate for parasite motions (low and high frequencies) of the carrier and to be able to orient the sensors in any direction. In addition, a forward motion compensation module equips each sensor to improve image quality. Without this, the image quality is degraded due to motion blur induced by the displacement of the carrier during the exposition of the sensor. The Sensor Block also contains an Inertial Navigation System (i.e., an Inertial Measurement Unit coupled with a GNSS sensor), whose goal is to precisely measure the position and orientation of the camera to determine accurately the position of pointed objects on the ground.
<figure><img src="image_113.png" /><figcaption>Figure 9.1: Simplified illustration of the on-board computer architecture in CAMELEON.</figcaption></figure>

- **On-board Computer**: it manages the data stream from the sensors to the User Interface and memory storage. Specifically, it consists of an *host* system that controls an FPGA programmed to read into the camera memory buffer and retrieves the images in the host memory. The host is then in charge of storing the images and their associated metadata in a geospatial database. It also feeds the user interface with the latest image for visualization. Finally, it also controls a set of lightweight GPUs (Nvidia Xavier or Orin) to execute complex treatments such as object detection. A special memory allocation allows for fast data transfer from the host to the kernel to process the image from the system in real-time. This is illustrated in Fig. 9.1, which depicts a simplified overview of the system's architecture.

- **User Interface**: it is an application that displays the images and metadata acquired by the system to the user. It also monitors a set of variables about the flight and the mission.

The Graphical Processing Units (GPUs) selected to be part of the system are the Nvidia edge computing devices Xavier and Orin (see Fig. 9.2). These are lightweight and power-efficient GPUs that can be used as an independent device or as an end-point within a more elaborate system. These GPUs have the computation capacity of mid-range commercial GPU (e.g., Nvidia RTX 3060) while consuming less than 50W, which is roughly four times as efficient. That makes them particularly well-suited for COSE application as only limited resources are available inside the carrier but a strong computation power is still required. Nevertheless, such an efficiency is not enough to run regular detection models in real-time on 100 of megapixels images. Thus, a lot of effort is required to adapt the models and their inference, we will elaborate this in Sec. 9.2.
<figure><img src="image_114.png" /><figcaption>Figure 9.2: Nvidia AGX Orin, development kit (left) and production (right) GPUs.</figcaption></figure>

The second component of the CAMELEON system is mainly constituted of the ground station. It is a high-power workstation, and its goal is to store and process data between missions. The low resources available during the flight are not sufficient for heavy treatments such as 3D modelization of the overflown areas. Mission disks filled during the flight inside the on-board segment can be transferred to the ground station for further analysis of the collected data. The ground station can also be used to execute more demanding detection algorithms or advanced treatments (e.g., segmentation, change detection, etc.).

### 9.1.2 CAMELEON Image Specifications and Constraints

In addition to the limited computation resource available on-board, CAMELEON has requirements that make object detection even more challenging. To assist the operators on-board efficiently, objects must be detected in real-time. The cameras are set to acquire at least one image every second, and each image is 11600 × 8700. These images are 16-bit Bayer matrices, which amount to roughly 192 megabytes per image. This means that more than 1 GB of data is produced every second in the system when all the cameras are used. However, the current prototype is designed with only one camera, which makes the real-time detection a little less challenging. Aside from the model inference, dealing with such a large amount of data requires carefully designed datastreams between the various parts of the system to retrieve, store and display the images produced by the cameras. This is what motivated the use of a Field Programmable Gate Array (FPGA) to orchestrate the transfer between the camera and the host.

Now, achieving real-time detection, i.e., processing one image per second, under such constraint is challenging and cannot be done with regular-size detection models (e.g., with ResNet backbone) implemented in Python. The fastest implementations in Pytorch (e.g., YOLOv5 [346]) are able to process one 640 × 640 pixels image in about 20ms on a high-end server GPU (Nvidia V100), with a backbone comparable in size to a ResNet-50. One image of CAMELEON is equivalent to 250 images640 × 640 pixels, which would require about 5s to process. Besides, this does not count data
transfer time, pre- and post-processing which must also fit under the one-second time limit. Fur-
thermore, the high-end GPU used for these benchmarks (available on YOLOv5's repository<fnref n="1" /> ) have
much more computing power than the Nvidia Xavier selected for CAMELEON (10 times more ac-
cording to the theoretical capabilities on Nvidia's website). Therefore, one must adapt the detection
models significantly to fulfill CAMELEON's requirements. This can be done in two different ways:
first reducing the model size and second accelerating the inference. Both approaches are employed
in CAMELEON's prototype, and they will be discussed in the following two sections.

9.2 Reducing Object Detection Model Size for Edge Computing

The most straightforward way to increase the throughput of a detection model is to reduce its size. However, this often comes at the cost of lower accuracy. In this section, we analyze to find the speed/accuracy tradeoff of YOLOv5 [346] on aerial images. Then we propose a simple knowledge distillation approach to improve this tradeoff and achieve higher detection quality at a fixed size.

9.2.1 Object Detection Accuracy/Speed Tradeoff

First, we compare the speed/accuracy tradeoff for multiple detection frameworks on natural im-
ages. The results of this comparison can be found in Fig. 9.3. To make this figure, we collected the
performance metrics and model size information directly from the articles presenting the various
detection methods. We select the mAP with various IoU thresholds on MS COCO. This is rather
simple as this is the most common evaluation benchmark in the detection literature. Then, the most
relevant way to assess the speed of the models is to measure the latency (in ms) which represents the
time required to process one image. However there are multiple complications with this measure:
it depends greatly on the hardware used, the size of the input images and even the version of the
library used during inference (in a less pronounced manner). In addition, the latency is not always
reported in the articles which makes the task even more difficult. Thus, we leverage a surrogate for
the latency: the total number of parameters in the model. Of course, it does not correlate fully with
the latency, but it is a sound approximation, and it is easier to collect. In Fig. 9.3, we plot the per-
formance against the number of parameters (left) and against the latency (right). There are missing
values in the latency plot as this measure was not reported in the original articles. We compare 8
distinct detection frameworks: YOLOv5 [346], CornerNet [43], CenterNet [44], Mask R-CNN [53],
DETR [59], FCOS [45], SwinTransformers [66] (based on Cascade R-CNN), DynamicHeads [73],
and DiffusionDet [74]. The conclusion of this comparison is blatant: YOLOv5 has a much better
speed/accuracy tradeoff than other detection frameworks. Of course, our comparison is not exhaus-
tive, yet it includes recent one-stage detectors that are fast and perform well. We are aware that
very recent developments in the YOLO family outperform YOLOv5 (e.g., YOLOv8 ²); however, we
could not include them in our comparison.

<footnote n="1">¹https://github.com/ultralytics/yolov5
²Link to YOLOv8 repository</footnote>
<figure><img src="image_115.png" /><figcaption>Figure 9.3: Inference speed / detection accuracy tradeoffs comparison for multiple detection frameworks on MS COCO. Latency, i.e., wall-clock time for inference of one image is directly used (right) but this measure is not always reported in the literature. As a surrogate, the number of parameters is used to have a more thorough overview (left). N, S, M, L and X designate the various model sizes.</figcaption></figure>

Another reason that nudges our choice toward YOLOv5 is the greater variety of model sizes that they propose. Most detection models are proposed with two different sizes, usually by employing two distinct backbones (e.g., ResNet-50 and ResNet-101), but keeping the detection head unchanged. With YOLOv5, the whole network is modified accordingly, including the head. This provides smoother model size modifications and greater flexibility.

The comparison from Fig. 9.3 demonstrates the superiority of YOLOv5 over other approaches on MS COCO. However, it is necessary to confirm that the same behavior is observed on aerial images as well. To this end, we train the different versions of YOLOv5 on DOTA and DIOR datasets and plot similar curves as in Fig. 9.3. The resulting plots are available in Fig. 9.4. In this analysis, we include even smaller models than the *nano* version of YOLOv5 (YOLOv5-N). We call these models YOLOv5-P and YOLOv5-F (for Pico and Femto following the nomenclature from YOLOv5). These models have respectively 0.68 and 0.32 millions of parameters which is much lower compared to traditional detection models. Nevertheless, strong performance is achieved on DOTA and DIOR datasets (see Tab. 9.1). Also, YOLOv5-X is not included in the analysis as it is too large for COSE's application.

From Fig. 9.4 and Tab. 9.1, it is clear that the detection performance is strongly correlated with the number of parameters of the model and that this connection holds also for very small models (< 1M parameters). However, it seems that the performance drops faster under a certain model size. This is observed both for DOTA and DIOR, even though it is more pronounced with DIOR. We also observed that the latency plot does not follow identically the parameters plot. For instance, with
<table><caption>Table 9.1: Performance (mAP<sub>0.5</sub>) on DOTA and DIOR for various YOLOv5 sizes. Numbers of parameters and latency are provided along with the performance measures. Latency is computed with 512× 512 images on a RTX 3090.</caption><thead><tr><td>Model</td><td># params</td><td>Latency (ms)</td><td>mAP DOTA</td><td>mAP DIOR</td></tr></thead><tbody><tr><td>YOLOv5-F</td><td>3.19e+5</td><td>5.2</td><td>42.8</td><td>41.4</td></tr><tr><td>YOLOv5-P</td><td>6.75e+5</td><td>5.4</td><td>49.1</td><td>63.2</td></tr><tr><td>YOLOv5-N</td><td>1.78e+6</td><td>6.0</td><td>68.5</td><td>80.3</td></tr><tr><td>YOLOv5-S</td><td>7.05e+6</td><td>6.6</td><td>72.7</td><td>85.2</td></tr><tr><td>YOLOv5-M</td><td>2.09e+7</td><td>8.7</td><td>74.6</td><td>88.1</td></tr><tr><td>YOLOv5-L</td><td>4.62e+7</td><td>11.1</td><td>75.4</td><td>89.1</td></tr></tbody></table>

<figure><img src="image_116.png" /><figcaption>Figure 9.4: YOLOv5 tradeoff between model size and performance (left). Between latency and performance (right). 6 model sizes are compared: Femto, Pico, Nano, Small, Medium and Large on DOTA and DIOR. The horizontal axis of the left plot is in log-scale to better distinguish between the smallest models. Latency is computed with 512× 512 images on a RTX 3090.</figcaption></figure>
YOLOv5-P, the latency does not follow the scaling down of the model size exactly. This is probably due to computation overhead and synchronization inside the model which prevents faster inference. Given these tradeoff curves, YOLOv5-N is the most promising model as it is closest to the top-left corner of the plots. However, it is relevant to investigate other model sizes and verify how well they comply with COSE's constraints.

### 9.2.2 Knowledge Distillation

Of course, the smaller the models, the lower the detection performance. However, there exist techniques to boost the performance of any network when we have access to a similar but larger model with increased performance. This is called *Knowledge Distillation* (KD). The main principle behind this technique is to train a *student model* to mimic a *teacher* which is often larger and has better performance. It was first introduced by Hinton et al. [347] in 2015. Originally, it consisted in training the student model with an additional loss measuring how close the logits from the student and the teacher are. Then, it was extended multiple times, with for instance intermediary layer activation distillation [348], relational distillation [349], adversarial distillation [350] or multiple teachers [351]. Most of these techniques are designed for classification applications and provide limited performance boosts for detection models. Fortunately, knowledge distillation can be extended for fine-grained tasks and in particular for object detection [352, 353, 354, 355, 356]. For more details about existing knowledge distillation methods, we defer the reader to this complete survey [357].

As a first try with KD, we applied Fine-grained Feature Imitation (FFI) [352] to improve the training of YOLOv5-P and YOLOv5-N with larger teachers trained on DOTA and DIOR. FFI proposes an additional loss function that measures the disparity between the teacher and student feature maps, but the computation only includes regions close to a ground truth annotation:

$$ \mathcal{L}_{FFI} = \frac{1}{2m} \sum_{i=1}^{W} \sum_{j=1}^{H} \mathcal{M}_{i,j} \| f_{\text{adapt}}(s_{i,j}) - t_{i,j} \|_{2}^{2}, \quad \text{where } m = \sum_{i=1}^{W} \sum_{j=1}^{H} \mathcal{M}_{i,j}. \qquad (9.1) $$

Here, $\mathcal{M}$ is the imitation mask which is 1 where in neighboring regions of each ground truth annotation and 0 elsewhere. $H$ and $W$ represent the height and width of the feature map respectively. $f_{\text{adapt}}$ is a mapping function that converts the feature of the student, denoted $s_{i,j}$, to the same size as the teacher's ones (in terms of the number of channels), denoted $t_{i,j}$. Indeed, features maps of the student often have fewer channel, which prevents direct comparison. The intuition behind this loss is that student outputs should only match teacher outputs in regions where there is an object of interest. The background is too noisy and the student model may not have the capacity to mimic the teacher everywhere, it should focus only in relevant regions.

The results obtained with aerial images do not agree with the performance gains reported in the original paper [352] for natural images (see Tab. 9.2). Significant performance drops are observed with distillation on aerial images. Similar drops are observed for YOLOv5-P and YOLOv5-N both on DOTA and DIOR datasets. KD methods designed for natural images may not be well adapted
<table><caption>Table 9.2: Detection performance (mAP<sub>0.5</sub>) comparison with and without Knowledge Distillation (KD). Performance is reported for different objects sizes and on two datasets, DOTA and DIOR. Both YOLOv5-P and YOLOv5-N are compared.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="4">DOTA</th><th colspan="4">DIOR</th></tr><tr><th>All</th><th>S</th><th>M</th><th>L</th><th>All</th><th>S</th><th>M</th><th>L</th></tr></thead><tbody><tr><td rowspan="2">YOLOv5-P</td><td>w/o KD</td><td>49.1</td><td>30.8</td><td>53.8</td><td>48.9</td><td>63.2</td><td>21.3</td><td>50.0</td><td>76.6</td></tr><tr><td>w/ KD</td><td>46.4</td><td>27.9</td><td>51.7</td><td>45.9</td><td>48.5</td><td>17.5</td><td>43.0</td><td>56.4</td></tr><tr><td rowspan="2">YOLOv5-N</td><td>w/o KD</td><td>67.9</td><td>51.0</td><td>70.9</td><td>69.8</td><td>80.6</td><td>35.9</td><td>69.2</td><td>90.8</td></tr><tr><td>w/ KD</td><td>47.5</td><td>30.3</td><td>52.4</td><td>45.5</td><td>55.9</td><td>22.0</td><td>51.4</td><td>65.3</td></tr></tbody></table>

for aerial images, one reason for this could be the presence of much smaller objects which have smaller and noisier representations inside the feature maps. In a sense, that could be linked to the difficulty of applying FSOD methods on aerial images, as described in Chap. 4. These results deserve to be investigated further in future work as distillation is a promising direction for detection improvements.

## 9.3 Inference Acceleration with TensorRT

TensorRT is a tool provided by Nvidia to optimize the inference of deep learning models on Nvidia's hardware. Recent Nvidia GPUs have dedicated modules for deep learning inference, called *Tensor Cores*. They contrast from *CUDA cores*, which are principally made for parallel computing. TensorRT unlocks the potential of the tensor cores by generating an *engine* that can be run most efficiently on a specific GPU. This differs from the vanilla Python inference (i.e., with any deep learning library), which calls CUDA kernels that are executed on the CUDA cores of the GPU. TensorRT also leverages the CUDA cores and thus leverages the maximum GPU capabilities. In addition, TensorRT proposes several optimization tricks for increasing the inference speed even more. We detail such tricks in the following paragraphs and explain how they can be used for object detection models.

Specifically, TensorRT takes as input a neural network model from any common library in a suitable format such as Open Neural Network Exchange (ONNX) and converts it into an *engine*. This engine contains the initial network along with specific instructions about how to run inference in an efficient way given the available hardware. TensorRT provides APIs for various programming languages (e.g., Python and C++) to run the engine directly from any application.

### Operation Fusion and Scheduling

TensorRT merges different layers and re-organizes the forward pass of the model. Specifically, the forward pass of a model can be represented as a computation graph (see Fig. 9.5). Each node represents a layer and the edges represent the data stream between layers. One of the objectives of TensorRT is to optimize the computation graph of the model. First, when several consecutive layers are composed (i.e., the output of the first layer is the only input of the second layer), they can be
<figure><img src="image_117.png" /><figcaption>Figure 9.5: Illustration of a computation graph and its optimization by TensorRT. Figure taken from an Nvidia technical blog post.</figcaption></figure>

fused vertically. Instead of creating distinct CUDA kernels for each layer, only one kernel is created, saving a lot of time, particularly in data transfer. This is illustrated in Fig. 9.5 (right) where, convolution, bias, and activation layers are merged as "CBR" blocks. Then, layers can also be merged horizontally. Horizontal fusion happens when similar operations are in parallel and have the same input (e.g., three distinct parallel $1 \times 1$ convolutions as illustrated in Fig. 9.5, right). This also reduces the creation of unnecessary CUDA kernels and maximizes the utilization of the resources. Finally, when there are parallel paths in the computation graph, the order in which the different branches can be irrelevant. In this case, some clever scheduling can maximize the utilization of the GPU and speed-up inference. This scheduling is not trivial as it is subject to the memory constraint of the hardware, and all parallel operations cannot be done at once.

TensorRT searches for optimal scheduling and fusion according to the available hardware and the input size that will be used at inference (e.g., batch size and image size). This implies that the input size is fixed for a given engine, but it is a small price compared to the benefits of the TensorRT conversion. TensorRT does have a variable size option, but it does not seem to be compatible with the reduced precision that we will detail in the following paragraphs.

## Floating point precision

Then, TensorRT reduces the precision of the model's weights and activations. In Python, common libraries store models and images as *float* numbers which take 4 bytes of memory (i.e., 32 bits, denoted F32). TensorRT can convert a model with half-precision floats (denoted F16), which only take 16 bits of memory. This reduces the size of the model by a factor of two, but most importantly it reduces the computation power required to perform a forward pass as well. Indeed, arithmetic operations are faster with half-precision numbers as they require fewer basic operations. Thus, it sensibly reduces the overall inference time. This trick is becoming quite common as well for training neural networks; however, mixed precision (i.e., keeping some parts of the models as regular floats) is required to prevent convergence issues [358]. TensorRT instead converts the whole model into
half-precision. In practice, this loss of information is not an issue for inference and performance is almost unchanged.

## Integer Quantization and Calibration

Pushing even further, TensorRT enables the conversion of the model in 8-bit integers. This again reduces the amount of memory and computation power required to store the model and perform inference. However, the model's weights and intermediary computation can only take 256 different values with integer precision. This is certainly not enough to represent the entire range of values found in a model. A solution for this is to map the weights and activations of the model into the $[-2^{b-1}, 2^{b-1}-1]$ interval, where $b$ is the number of bits used for the quantization (generally $b = 8$). If the dynamic range of the weights is $[\alpha, \beta]$, this can be achieved with a linear transformation:

$$S(x) = ax + b, \quad \text{with } a = \frac{2^b - 1}{\beta - \alpha}, \quad \text{and } b = -\frac{\alpha(2^b - 1)}{\beta - \alpha} - 2^{b-1}. \qquad (9.2)$$

Hence, the quantization and dequantization functions can be written as:

$$\bar{x} = Q(x) = \lfloor S(x) \rfloor = \lfloor ax + b \rfloor, \qquad (9.3)$$

$$x \approx S^{-1}(\bar{x}) = \frac{1}{a}(\bar{x} - b). \qquad (9.4)$$

This can be further simplified when both the weights interval and the quantized interval are centered (e.g., $\alpha = \beta$): $S^{-1}(\bar{x}) = \frac{1}{a}\bar{x}$. In this case, one can easily see that the matrix multiplication, the basic operation of neural networks' forward pass can be computed using mostly integer multiplications and additions. If we define three matrices $X \in \mathbb{R}^{n \times p}$, $Y \in \mathbb{R}^{p \times m}$, and $Z \in \mathbb{R}^{n \times m}$, such that $Z = XY$, then we have:

$$z_{i,j} = \sum_k x_{i,k} y_{k,j} \approx \sum_k S_x^{-1}(\bar{x}_{i,k}) S_y^{-1}(\bar{y}_{k,j}) = \frac{1}{a_x a_y} \sum_k x_{i,k} \bar{y}_{k,j}, \qquad (9.5)$$

where $a_x$ and $a_y$ are the quantization scale parameters for the respective quantization of matrices $X$ and $Y$. As hinted in the previous equation, different quantization functions are necessary to keep the flexibility of the model. The number of parameters inside a model is large compared to the number of quantized values (256 for integer quantization). Quantizing all weights of the model at once would result in a much looser approximation of the actual weight values. Instead, quantization is performed at the layer-level or even lower (at the column or channel level).

Similarly, activations of the model are also quantized per layer. However, the range of the activation highly depends on the input of the model and cannot be known in advance. To alleviate this, a calibration cache can be computed from a calibration set. TensorRT automatically builds this cache by running a forward pass on all images of the calibration set. Of course, this set must contain
images that are similar to the images on which the model will be deployed. Concerning COSE, this
complexifies the domain adaptation problem as the calibration of the model should be adapted to
the domain. Even if it only requires non-annotated images to compute the calibration cache, it can
sometimes be challenging to obtain an appropriate calibration set due to confidentiality constraints.
The resulting calibration cache gathers information about the quantization of the activation of the
different layers of the model. It is then used during inference, which slightly increases memory
usage during inference.

We briefly presented the linear quantization technique, yet TensorRT also provides more elaborated quantization to minimize the loss of information. For instance, it has an Entropy Calibration technique that adapts the density of bins according to the density of weights values. It is very similar to entropy coding techniques that assign smaller code to the most frequent symbols.

C++ Implementation

Finally, the TensorRT conversion allows for using C++ as a backend instead of Python. This is not strictly speaking a TensorRT trick, but it significantly reduces the inference time. As TensorRT provides a C++ API, the whole inference pipeline can be written in C++ as well. C++ is known to be much faster than Python and implementing pre- and post-processing in this language can speed up the inference.

9.4 Object Detection Pipeline for CAMELEON Prototype

Now that we have discussed various improvements for accelerating the inference time of the detec-
tion model, we present how we make use of them specifically for COSE application. First, we select
the most interesting models from our analysis in Sec. 9.2: YOLOv5-N and YOLOv5-P. Both achieve
fast inference while preserving high detection performance. We then perform the conversion in a
TensorRT engine to speed up the inference. This was done only for the model trained with DOTA
dataset, but it would be identical with DIOR.

9.4.1 Deployment with TensorRT

We started with the nano version as the goal is to satisfy the time constraint of processing one image
per second. If it does not fit under the time constraint, we would do the same with the pico version.
With the help of TensorRT, we reduce the precision of the model into 8-bits integers. The calibration
is performed with DOTA validation images. However, to find the most efficient utilization of the
GPUs, it was necessary to explore what were the best combinations of batch size and input image
size. CAMELEON's images have 11600 × 8700 pixels and doing the inference on the entire image
at once demands too much GPU memory, even with our smallest models. The images must be tiled
before being fed to the model. The tiling has a major downside: it can cut object in half which make
them more difficult to detect. Adding some overlapping between tiles can prevent this but it also
increases exponentially the amount of data to process (see Fig. 9.6). In addition, having to many tiles
requires processing multiple batches. This is equivalent to performing the inference multiple times
<figure><img src="image_118.png" /><figcaption>Figure 9.6: Show the number of tiles against the overlap, for different tile size (left) and the ratio of pixels that need to be processed given the overlap (right). As an example, choosing an overlap of 0.7 will produce 10 times more data to process compared with the original image. The computations were done with CAMELEON image size (11600 × 8700 pixels).</figcaption></figure>

adding a small overhead at each iteration. Thus, we want to minimize the overlap and have tiles as large as possible.

We tried several combinations of batch size and tile size, but this is a tedious task as an entire engine must be created for each setting. We did not mention it before, but the engine generation is a heavy optimization process that requires several hours on the Xavier GPUs. Based on these insights and multiple practical trials, we set the tile size to half the height of a CAMELEON image: 4350, with a batch size of 1. This maximizes the utilization of the GPU, requires few inferences and limits object splitting. This results in 6 tiles organized as shown in Fig. 9.7 with slight horizontal overlap.

Overlapping also induces the need for NMS after the detection. Indeed, in areas that will be pro-
cessed multiple times, detections can be duplicated. To remove these undesired boxes, an NMS
operation must be performed at the whole image level. As it scales in O(M²) with M the number
of detected boxes, it can become expensive. In addition, it depends on the number of detected ob-
jects in the image, which can be considerable in CAMELEON-size images. Fortunately, there exist
fast GPU implementations of the NMS that can be leveraged easily. It could be further optimized
by first performing the NMS on each tile individually and then only in the overlapping areas to
drastically reduce the number of boxes. Nonetheless, the NMS computation time is not prohibitive
as it is and the test images contain far more objects than what will be encountered in real-case
applications. Therefore, we choose not to spend effort on improving the NMS process. The pre-
and post-processing are less expensive than the inference itself, but they still consume a significant
amount of time. To accelerate them, we choose to implement them in C++ as it is much faster than
Python, especially for data management and transfer.
<figure><img src="image_119.png" /><figcaption>Figure 9.7: Illustration of the tiling implemented in our prototype, the background image is a test image, it is constituted of DOTA images pasted together to get the right dimensions. Green squares represent the tiles.</figcaption></figure>
<table><caption>Table 9.3: Profiling for the inference of YOLOv5-N engine with integer precision on one CAMELEON image. Timings are measured on a Nvidia AGX Xavier.</caption><thead><tr><th>Processing step</th><th></th><th>Time (ms)</th></tr></thead><tbody><tr><td rowspan="2">CPU to GPU copy</td><td></td><td>87 ± 0.5</td></tr><tr><td>Tiling</td><td>1 ± 0.1</td></tr><tr><td rowspan="3">Inference per tile<br/>×6</td><td>Pre-processing</td><td>24 ± 3</td></tr><tr><td>Inference</td><td>73 ± 1</td></tr><tr><td>GPU to CPU copy</td><td>10 ± 0.5</td></tr><tr><td></td><td><strong>Subtotal</strong></td><td><strong>642</strong></td></tr><tr><td>Post-processing</td><td></td><td>16 ± 1</td></tr><tr><td><strong>Total</strong></td><td></td><td><strong>745 ± 10</strong></td></tr></tbody></table>

### 9.4.2 Detection Performance Evaluation and Profiling

Now that we have a TensorRT engine, we must validate its capacity, both in terms of inference speed and detection quality. Indeed, the precision reduction of the model often leads to reduced performance.

First, we check if the inference is fast enough to comply with COSE's constraints. To this end, a simple profiling of the various steps of the execution is realized. Specifically, we isolate the data copy, tiling, pre-processing, inference and post-processing. The results of this profiling can be found in Tab. 9.3. The timings are averaged over 100 runs on an Nvidia AGX Xavier in high consumption mode (30W + overclocking). It results in the overall processing of one image in roughly 750ms. This falls under the 1s barrier and thus fulfills the application constraints. Larger versions of YOLOv5 do not respect this constraint. However, it would be relevant to investigate smaller ones to get degraded performance mode to process more than one image per second. The current prototype has only one sensor, but the final system will fly with 6 identical cameras. While it may not be realistic to process all these images in real-time it might be useful to have access to faster models able to manage the image from more than one sensor. We also experiment with a new generation of embedded GPU: the Nvidia AGX Orin, which recently replaced the Xavier. Compatibility issues between TensorRT and the new GPU prevent us from experimenting more with it, but we were able to create a similar engine for the Orin. This produces significant inference speed gains as we process an entire image in a little less than 500ms. Yet, the Orin consumes more power than the Xavier (50W).

To assess how the precision reduction changes both the inference speed and detection performance, we compare multiple deployment strategies with the YOLOv5-N model. Specifically, we compare the inference speed of the Pytorch model (with F32 and F16 precisions) with the TensorRT engines (F32, F16 and INT8 precisions). The inference times are measured both on an RTX 3090 and an Nvidia AGX Xavier, with distinct image widths: 512 and 4096 pixels. The results of this comparison
<table><caption>Table 9.4: Inference speed comparison (in ms) between various deployment strategies and precisions for YOLOv5-N. Inference times are measured with two image widths and on two GPUs. A high-end desktop GPU: RTX 3090, and an embedded GPU: AGX Xavier. mAP with a 0.5 IoU threshold is also reported on DOTA.</caption><thead><tr><th rowspan="2">Image Size (pixels)</th><th colspan="2">RTX 3090 Latency</th><th colspan="2">AGX Xavier Latency</th><th rowspan="2">mAP<sub>0.5</sub></th></tr><tr><th>512</th><th>4096</th><th>512</th><th>4096</th></tr></thead><tbody><tr><td>Pytorch F32</td><td>6</td><td>36.3</td><td>87.5</td><td>2638.6</td><td>0.679</td></tr><tr><td>Pytorch F16</td><td>8.3</td><td>20.4</td><td>90.7</td><td>2616.5</td><td>0.679</td></tr><tr><td>TensorRT F32</td><td>0.75</td><td>17.16</td><td>4.65</td><td>199.22</td><td>0.565</td></tr><tr><td>TensorRT F16</td><td>0.44</td><td>7.22</td><td>4.64</td><td>106.2</td><td>0.565</td></tr><tr><td>TensorRT INT8</td><td>0.43</td><td>5.77</td><td>2.48</td><td>68.65</td><td>0.523</td></tr></tbody></table>

are available in Tab. 9.4. First, there is an impressive speed gap between the TensorRT engines and Pytorch models: TensorRT engines are much faster. This is expected given all the optimizations conducted and the relative slowness of Python. However, it is worth noting different behaviors between the two GPUs. Of course, the RTX 3090 is faster, but changing the engine precision from F32 to F16 does not produce similar gains for the AGX Xavier. This is explained by the change of microarchitecture between these two GPUs. The AGX Xavier is based on the Volta architecture while the RTX 3090 leverages the newer Ampere architecture. In particular, Ampere introduces a new generation of Tensor core, specifically designed for F32 computations. Reducing the precision of the models generates a slight overhead and higher gains are observed when the GPU utilization is higher, i.e., when using larger images or increased batch size. This is why we report the results both for 512 and 4096 image widths. Inference gains are more important with larger image sizes, fortunately for our application. Then, we also report the mAP for each model. We observe no performance drop when switching from F32 to F16. But, there is a slight decrease of mAP with the quantization in INT8. This is expected as a lot of approximations are made in the computations. On the other hand, we also observe a performance drop with the TensorRT conversion, with the F32 engine this is not expected as no approximation should be done. While the drop is acceptable, it should be investigated in depth to better understand its origin. Finally, we report some qualitative results of the INT8 inference on a CAMELEON-size image in Fig. 9.8. The inference on the whole image was performed in less than a second on an AGX Xavier.

More experiments are planned in future work to find an even better compromise between inference speed and detection quality, especially with the Orin GPUs. However, the compromise attained with YOLOv5-N is already satisfactory from an industrial perspective and will be integrated into the first CAMELEON prototype. The connection with the CAMELEON user interface Fig. 9.9 has already been done and the models wait for the first flight to be tested on real images.
<figure><img src="image_120.png" /><figcaption>Figure 9.8: Qualitative detection performance on CAMELEON-size image with YOLOv5-N INT8 engine.</figcaption></figure>

<figure><img src="image_121.png" /><figcaption>Figure 9.9: CAMELEON user interface.</figcaption></figure>
9.5 Conclusion

In this chapter, we have presented some tools and tricks that are leveraged to greatly accelerate
the inference of neural networks. Thanks to these tools, we are able to achieve real-time detection
on images of hundreds of megapixels, embedded on a low-consumption GPU. Of course, better
compromises could be found with a more complete study of the YOLOv5 framework and especially
by exploring the influence of the model size on the performance with greater granularity. Aside from
finding a better compromise, it would be relevant to investigate deeper knowledge distillation and
 unlock its full potential for object detection. It is also required to explore new detection frameworks
(e.g., YOLOv8), in particular with the recent AGX Orin. Its improved computation power could
allow for larger model sizes and increased performance while satisfying the time constraint. Finally,
this deployment has been conducted for regular detection models, it should be done as well for Few-
Shot detection models, even though it complexifies greatly the analysis and the creation of TensorRT
engines. While it is unrealistic to deploy attention-based models in this manner (due to increased
inference time), fine-tuned models are much more adapted. It would increase the adaptation time
by adding the engine creation after fine-tuning, but the inference will remain unchanged. Of course,
it would be necessary to study the influence of the model size on the performance and see if that is
compatible with CAMELEON's requirements.
# CONCLUSION AND PERSPECTIVES

In a nutshell, we first summarize the various contributions made to the Few-Shot Object Detection field. Then, we take a step back from these contributions and discuss what are the most promising research directions that should be followed in future work. Finally, we present the remaining industrial challenges that COSE will face before achieving robust and adaptable object detection in an embedded environment.

## Contribution Summary

In the first part of the manuscript we have reviewed thoroughly the literature about Object Detection, Few-Shot Learning and especially Few-Shot Object Detection. These three domains are the foundations of this PhD project. The knowledge presented in the corresponding chapters is crucial for the development of future FSOD algorithms and to meet the industrial needs of COSE. These three fields are growing rapidly and it is necessary to remain up-to-date with state-of-the-art both for academic and industrial research. It is especially important for FSOD as it is a very recent problem and lacks consensus on how to address it. Even if the FSOD literature is growing, it is still a small field and most contributions are primarily designed for natural images. However, it is not guaranteed that such techniques will perform well on other kinds of images. In Chap. 4, we highlight especially that with an in-depth analysis of the FSOD performance on aerial images, the performance drops significantly when the methods are applied to these kinds of images. The main reason behind this phenomenon is the smaller object size in aerial images. Even if small objects are already difficult for regular object detection, the challenge is much greater in the few-shot regime. In addition, Part I highlights the organization of the FSOD literature into three kinds of approaches. We divided Part II into three chapters accordingly, each focusing on a different FSOD approach: metric learning, attention-based methods and fine-tuning strategies. Specifically, in Chap. 5, we have proposed an original FSOD method entirely based on metric learning. It embeds prototypical networks into the well-known Faster R-CNN detection model. This naive approach achieves mitigated results but is highly adaptable, it can adapt to novel classes without fine-tuning or heavy computation. The experiments conducted with this model are instructive for the development of future FSOD methods. Then, in Chap. 6, we focused on the attention-based mechanisms for FSOD. We