k-means does not produce uniformly distributed indices except in singular cases [Gray, 1998].

**Entropy model.** To improve the bitrate, we hence rely on an entropy coder, which provides an inverse pair of functions, enc_p and dec_p, achieving near optimal compression rates on sequences of symbols for any distribution p. The better p matches the (unknown) underlying data distribution, the better the compression rate [Cover, 1991]. Generally, more powerful generative models will ensure better compression performance.

Fully autoregressive generative models $p(\mathbf{q}) = \prod_{t=1}^{T} p(q_t|\mathbf{q}_{<t})$ are powerful [Ding, 2021a; Esser, 2021; Gafni, 2022; Ramesh, 2021a; Yu, 2021], however, they are inconvenient in that the likelihood estimation for this type of model is not trivially parallelizable: each patch index must be processed sequentially as it is used to condition subsequent patch indices. Thus, similar to prior works [Chang, 2022; Reed, 2017] we propose a masked image model, which we use to predict the image patch indices in several stages. Specifically, we partition $\mathbf{q}$ into $S$ subsets $\mathbf{q}_1, \mathbf{q}_2, \dots, \mathbf{q}_S$ of patch indices, that we refer to as *tokens* by analogy to language modelling:

$$ \mathbf{q} = \bigcup_{s=1}^{S} \mathbf{q}_s. \qquad (5.1) $$

We model the elements in each subset conditionally independent given all preceding groups:

$$ p(\mathbf{q}) = \prod_{s=1}^{S} p(\mathbf{q}_s|\mathbf{q}_{<s}; \theta_s), \qquad (5.2) $$

$$ p(\mathbf{q}_s|\mathbf{q}_{<s}; \theta_s) = \prod_{q_t \in \mathbf{q}_s} p(q_t|\mathbf{q}_1, \mathbf{q}_2, \dots, \mathbf{q}_{s-1}; \theta_s). \qquad (5.3) $$

Since $p(\mathbf{q}_1)$ is not conditioned on any previous elements, it fully factorizes over $q_t \in \mathbf{q}_1$, and we model it as the marginal distribution over the vocabulary observed on the training data. The non-trivial conditional distributions $p(\mathbf{q}_s|\mathbf{q}_{<s}; \theta_s)$ for $s \ge 2$ are modeled using transformer networks, which have few inductive biases and have been successful across many tasks, including image generation. An overview of the MIM entropy model is illustrated in Figure 5.2.

**Amortized encoding and decoding.** From a computational point of view, our proposal allows for the compression (resp. decompression) to proceed in $S$ stages. In each stage $s$ we encode/decode the set $\mathbf{q}_s$ of tokens conditioned on the groups $\mathbf{q}_1, \dots, \mathbf{q}_{s-1}$ encoded/decoded in preceding stages, but independently among the tokens in the set $\mathbf{q}_s$. This allows for parallelization among the elements in each subset $\mathbf{q}_s$, and requires strictly $S$ forward passes through the model independent of the image size, rather than $T$ sequential forwards passes for fully autoregressive models.

**Quincunx partitioning.** In practice, we typically use $S = 5$ stages. We have explored different patterns to partition the $T$ tokens over the $S$ stages. In particular we consider the “quincunx” regular grid pattern, where in each stage we double the number of tokens to predict, see Figure 5.4 for an illustration. This multi-level refinement was previously explored for image compression in the context of lifting schemes designed with oriented wavelets [Chappelier, 2006]. In our experiments we contrast this partitioning with alternative ones with other patterns and
<figure><img src="image_24.png" /><figcaption>Figure 5.5 – Rate-Distortion performance for Tecnick and Kodak datasets. We report PQ-MIM PSNR and MS-SSIM performance for various operating points. PQ-MIM provides a competitive performance, particularly for MS-SSIM, compared to standard codecs such as JPEG 2000 [Taubman, 2012] and BPG [Bellard, ] as well as recent neural methods [Cheng, 2020; Minnen, 2018; Ballé, 2018].</figcaption></figure>

subset cardinalities (Figure 5.9).

5.3.3 Training the PQ-MIM

**Reconstruction objective and training.** The goal of lossy image compression is to match the image and its reconstruction as closely as possible according to some distortion metric. In this chapter, we train our model to reduce the image distortion and the quantization loss using the following objective:

$$
L = L_{\text{Rec}} + \eta \cdot L_{\text{PQ}} \qquad (5.4)
$$

Following VQ-VAE, our quantization objective $L_{\text{PQ}}$ consists of embedding and commitment losses, averaged over the $M$ different PQ sub-vectors. For the distortion loss $L_{\text{Rec}}$, we present two setups where we use different types of distortion measures:

— MSE & MS-SSIM. Typical distortion measures used in the majority of the neural compres-
sion literature such as mean squared error (MSE) or multi-scale structural similarity [Wang,
[2003] (MS-SSIM). For this setup, the model is trained solely using one distortion measure at a time.

$$L_{\text{Rec}}(\mathbf{x}, \hat{E}, G) = L_{\text{MSE/MS-SSIM}}(\mathbf{x}, \hat{\mathbf{x}}) \qquad (5.5)$$

— **Perceptual measures.** Alternatively, we report a setup where we utilize perceptual objectives such as LPIPS [Zhang, 2018] and adversarial training [Goodfellow, 2014] to enhance psycho-visual image quality. The distortion loss is defined as:

$$L_{\text{Rec}}(\mathbf{x}, \hat{E}, G) = L_{\text{MSE}}(\mathbf{x}, \hat{\mathbf{x}}) + \alpha \cdot L_{\text{Perc}}(\mathbf{x}, \hat{\mathbf{x}}) + \gamma \cdot L_{\text{Adv}}(\hat{E}, G, D), \qquad (5.6)$$

where $\alpha$ and $\gamma$ are weighing coefficients and the adversarial loss $L_{\text{Adv}}$ is defined as:

$$L_{\text{Adv}}(\hat{E}, G, D) = \mathbb{E}_{\mathbf{x}}[\ln D(\mathbf{x})] + \mathbb{E}_{\hat{\mathbf{x}}}[\ln(1 - D(\hat{\mathbf{x}}))], \qquad (5.7)$$

where $D(\cdot)$ is the discriminator and $\mathbb{E}_{\mathbf{x}}$ denotes the expectation over $\mathbf{x}$ sampled uniformly from the training set. Similarly $\mathbb{E}_{\hat{\mathbf{x}}}$ denotes the expectation over reconstructed training images. Note that, unlike the MSE and perceptual losses, the adversarial loss does not compare individual images and their reconstructions, but aims to match the distributions of original images and their reconstructions.

**Training the entropy model.** Our MIM module is an XCiT transformer that accepts $T$ tokens as input representing the image patches. During training, we randomly mask a set of tokens by sampling from a uniform distribution $U(0, 1)$. The masked tokens are replaced with a mask embedding vector, while the observed token indices are mapped to their corresponding continuous representation using an embedding look-up table. The MIM module outputs a context vector for every masked token which in turn is passed to $M$ linear heads, representing the different PQ sub-vector indices, followed by a softmax to yield a distribution $p(\mathbf{q}_s|\mathbf{q}_{<s})$. The module is trained using a standard cross-entropy objective. While we train the autoencoder and the MIM modules simultaneously, we do not backpropagate gradients from the MIM module to the encoder $E$ or the quantization parameters, so the two components can be trained separately in sequence.

## 5.4 Experiments

We first present our experimental setup in Section 5.4.1 and the present our results in Section 5.4.2. We provide ablation studies in Section 5.4.3. We will share our code and models.

### 5.4.1 Experimental setup

**Rate-distortion control.** For all our experiments we fix the codebook size $V = 256$ and only vary the number of sub-vectors $M \in \{2, 4, 6\}$ for two different down-sampling factors $f \in \{8, 16\}$. **PQ-VAE implementation details.** Our PQ-VAE training uses the straight-through estimator [Bengio, 2013] to propagate gradients through the quantization bottleneck. As for the quantization, the elements of the latent representation $\mathbf{z}$ are first linearly projected to a low
<figure><img src="image_25.png" /><figcaption>Figure 5.6 – **Perceptual training and evaluation using CLIC 2020 test-set.** Performance of our adversarially trained PQ-MIM w.r.t perpetual metrics compared to HiFiC [Mentzer, 2020]. PQ-MIM provides a stronger performance on FID, KID and MS-SSIM across all reported operating points.</figcaption></figure>

dimensional look-up vector (dim=8 per sub-vector) followed by $l_2$ normalization, following Yu *et al.* [Yu, 2021]. We train our model using ImageNet [Deng, 2009a] for 50 epochs with a batch size of 256. We use an AdamW [Loshchilov, 2017] optimizer with a peak learning rate of $1.10^{-3}$, weight decay of 0.02 and $\beta_2 = 0.95$. We apply a linear warmup for the first 5 epochs of training followed by a cosine decay schedule for the remaining 45 epochs to a minimum learning rate of $5.10^{-5}$. Unless mentioned otherwise, for all experiments, the encoder and decoder use an XCiT-L6 with 6 layers and a hidden dimension of 768. We use sinusoidal positional embedding [Vaswani, 2017a] such that our model can flexibly operate on variable-sized images.

For the results reported in Figure 5.5, the models are trained using solely MSE ($\eta = 0.5$) or MS-SSIM ($\eta = 10.0$) distortion losses for their corresponding plots. As for the models trained with perceptual objectives (Figure 5.6 and Table 5.1), they are trained with a weighted sum of MSE, LPIPS ($\alpha = 1$) and adversarial loss ($\gamma = 0.1$). We use a ProjectedGAN Discriminator [Sauer, 2021] architecture. The perceptual training is initialized with an MSE only trained checkpoint and trained for 50 epochs using ImageNet with a learning rate of $10^{-4}$ and weight decay of $5.10^{-5}$. Similar to HiFiC [Mentzer, 2020], we freeze the encoder during the perceptual training. Additionally, we find that clipping the gradient norm to a maximum value of 4.0 improves the training stability. Our discriminator takes only the decoded image as input and does not rely on any other conditional signal.

**MIM implementation details.** Our MIM module is an XCiT-L12 with 12 layers and embedding dimension of 768. MIM is trained simultaneously with the PQ-VAE, but the gradients are not backpropagated to the encoder or the quantizer parameters. The PQ indices $q$ are split into inputs and targets for the MIM model as defined by the quincunx partitioning pattern. By default we use $S = 5$ stages. All stages are processed with the same MIM model. The masked patches are replaced by a learnable “mask” token embedding. The loss is computed only for the masked patches. Since every token is assigned $M$ PQ indices, the output of the MIM transformer is passed to $M$ separate linear heads to predict $M$ softmax normalized distributions over their corresponding codebooks. The marginal distribution of the codebook is computed as a normalized histogram over the ImageNet training set. For entropy coding, we use the implementation of the `torchac`<fnref n="1" /> arithmetic coder.

<footnote n="1">1. https://github.com/fab-jul/torchac</footnote>
Chapter 5. Image Compression with Product Quantized Masked Image Modeling

<figure><img src="image_26.png" /><figcaption>Figure 5.7 – Extreme Image Compression. PQ-MIM exhibits non-trivial compression performance at the extreme compression regime (e.g. 0.03 bpp), leading to compressed image codes that can fit in a short tweet (280 characters).</figcaption></figure>

**Datasets.** We train our models using ImageNet [Deng, 2009a]. For data augmentation, we apply random resized cropping to 256×256 images and horizontal flipping. For evaluation and comparison to prior work, we use KODAK [Kodak, 1993] and TECNICK [Asuni, 2014] datasets for PSNR and MS-SSIM. Moreover, we compute the perceptual metrics (FID [Heusel, 2017], KID [Bińkowski, 2018]) for perceptually trained models using the CLIC 2020 test-set [Toderici, 2020] (428 images) using the same patch cropping scheme detailed by [Mentzer, 2020].

**Baselines.** We compare to several existing neural compression baselines: the scale hyperprior model [Ballé, 2018], mean & scale hyperprior [Minnen, 2018], GMM hyperprior [Cheng, 2020], and HiFiC [Mentzer, 2020]. Among the non-neural codecs, we compare to the popular BPG [Bellard, ], WebP, and JPEG2000.

5.4.2 Main experimental results

**Comparison to existing (neural) codecs.** We compare PQ-MIM to other approaches across a wide range of bitrates in Figure 5.5². Note that in our evaluation we consider bitrates that are an order of magnitude lower than what is typically studied in the literature: most previous studies were limited to 0.1 bpp and above, see e.g. [Ballé, 2018; Cheng, 2020; Minnen, 2018]. The extremely low bitrates we consider make it possible to transmit a 256×256 image in an SMS or a tweet (280 characters)³ as shown in Figure 5.7. PQ-MIM achieves a strong and competitive performance for both KODAK and TECNICK datasets, outperforming all prior neural and standard codecs with the exception of GMM hyperprior [Cheng, 2020]. We observe that PQ-MIM is particularly strong for low rates, making it a good fit for extreme compression scenarios. Moreover, PQ-MIM exhibits a particularly strong MS-SSIM performance which was designed to model the human visual contrast perception [Wang, 2004; Wang, 2003].

<footnote n="2">2. All results for the baselines are reported using the authors' official repositories
3. For example, a bitrate of 0.03 yields 256 × 256 × 0.03/8 ≈ 246 bytes for a 256×256 image.</footnote>
<table><caption>Table 5.1 – Discriminator Architecture. We investigate multiple discriminators including Style-GAN [Karras, 2019], ProjectedGAN [Sauer, 2021] and UNet [Schönfeld, 2020].</caption><thead><tr><th>Discriminator</th><th>FID ↓</th><th>KID ↓</th><th>MS-SSIM ↑</th></tr></thead><tbody><tr><td>None</td><td>26.1</td><td>1.2×-2</td><td>15.3</td></tr><tr><td>StyleGAN</td><td><strong>3.57</strong></td><td><strong>4.8×10<sup>-4</sup></strong></td><td>13.3</td></tr><tr><td>ProjectedGAN</td><td>3.69</td><td>6.6×10<sup>-4</sup></td><td><strong>14.1</strong></td></tr><tr><td>UNet</td><td>3.87</td><td>6.7×10<sup>-4</sup></td><td>13.9</td></tr></tbody></table>

<table><caption>Table 5.2 – Different MIM masking policies. PQ-MIM with quincunx pattern with 5 steps reduces the bpp significantly (27%). Additionally, PQ-MIM is orders of magnitude cheaper in terms of FLOPs compared to an autoregressive raster order masking pattern.</caption><thead><tr><th>Masking policy</th><th>#steps</th><th>bpp</th><th>MACs/Pixel (M)</th></tr></thead><tbody><tr><td>Marginal baseline</td><td>1</td><td>0.512</td><td>0.69</td></tr><tr><td>Raster order</td><td>T</td><td>00M</td><td>24.4×10<sup>3</sup></td></tr><tr><td>Quincunx</td><td>5</td><td>0.373</td><td>6.66</td></tr></tbody></table>

**Perceptual metrics comparison.** In Figure 5.6, we compare PQ-MIM to HiFiC [Mentzer, 2020] in perceptual quality measures like FID and KID as well as MS-SSIM. HiFiC is based on the mean & scale hyperprior model [Minnen, 2018], but adds an adversarially trained discriminator model to improve the perceptual quality of the image reconstructions. PQ-MIM, with perceptual training, achieves a strong performance for all reported metrics, outperforming HiFiC for all operating points.

### 5.4.3 Analysis and Ablations

**Model size and architecture.** In Figure 5.8, we analyze the effect of using XCiT of different capacities for the autoencoder with respect to rate-distortion trade-off. We observe that the performance improves with higher capacity autoencoders, but their is a diminishing return with a further increase in capacity. For all our experiments we use an XCiT-L6 since it achieves the best performance.

**Masking patterns.** In Table 5.2, we compare to predicting tokens one-by-one autoregressively in a raster-scan order, the same pattern used in VQ-VAE based generative image models such as DALL-E [Ramesh, 2021a] and VQ-GAN [Esser, 2021]. In contrast to PQ-MIM, raster-scan models require causal attention, which makes XCiT not a good fit. We use a standard ViT model instead. However, due to the quadratic complexity of ViT and the high resolution of images typically used for evaluation of compression method (e.g. TECNICK), our autoregressive variant consistently exceeded the memory limits, even when using A100 GPUs with 40GB memory. Moreover, raster scan fully-autoregressive models results in extremely expensive FLOP count since it needs *T* separate forward passes per image. On the other hand, our stage-wise MIM with quincunx pattern requires 4 evaluations, and does not scale with the image resolution as is the case for raster, making it a more practical solution.

**Number of prediction stages.** We compare the quincunx masking pattern with masking based on the confidence score following MaskGiT [Chang, 2022] patch selection procedure for image generation. In the latter case, at a given step, we pick the patches to transmit dynamically based on their confidence score. The confidence is defined as the maximum across the probabilities assigned over the vocabulary by the model.<fnref n="4" /> For the quincunx and confidence-based

<footnote n="4">4. Note that for decoding the same confidence score can be used to identify the group of tokens to decode.</footnote>
<figure><img src="image_27.png" /><figcaption>Figure 5.8 – **Autoencoder capacity.** The RD performance of different encoder/decoder capacities. We use the same model trunk for (i) the encoder; (ii) the PQ-MIM entropy model; and (iii) the decoder. Increasing the model size from an XCI-T-6 model (3.5M params) to XCI-T-L6 (47M params) increases the performance by typically +0.8dB.</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>Figure 5.9 – **MIM number of steps.** In addition to quincunx, we explore using the prediction confidence as the patch masking policy following MaskGiT [Chang, 2022]. We test a linear and doubling schedules. Quincunx provides a higher rate saving, even compared to a confidence policy with a longer schedule.</figcaption></figure>

masking policy we use the same 5-stage scheme, in which the number of tokens in subsequent
groups doubles in size. We ablate the number of prediction stages *S* for the quincunx and
confidence-based sampling. For the latter we consider two options: (i) a linear scheme where
each group of tokens contains (approximately) the same number of tokens; and (ii) a doubling
scheme where each subsequent group of tokens is double the size of the previous group, as is
also used for the quincunx pattern. Every point on the curves in Figure 5.9 corresponds to the
bitrate when encoding/decoding with a given number of steps. For example, *S* = 2 steps means
we have only two steps each encoding/decoding 50% of the patches. For the 3-steps doubling
schedule the groups have sizes of 1/4, 1/4, 1/2, and so on. For all three patterns the bitrate
monotonically decreases with the number of steps, as more tokens can be predicted from larger
contexts.

On the one hand, using confidence-based masking patterns, doubling and linear schemes lead to mostly comparable bitrates for the same number of steps, with further improvement for linear with more steps however with diminishing return. On the other hand, quincunx provides a stronger reduction ratio with strictly 5 steps, even when compared with confidence-based masking with a higher number of steps.

**Predicting missing patches.** When encoding/decoding the discrete image representation *q*, we reduce the bitrate with our models by their ability to predict the remaining tokens given those of preceding stages. To illustrate the predictive ability of our model, we consider an experiment where we remove a subset of the tokens (sampled randomly), and use our model to fill in the missing patches by conditioning on the observed ones. We then use the PQ-VAE decoder to decode the discrete latent codes including the filled-in ones. In Figure 5.10 we show the results obtained when removing 10%, 20%, 30% and 50% of the patches. MIM can model
<figure><img src="image_29.png" /><figcaption>Figure 5.10 – PQ-MIM can operate on a partial set of transmitted tokens. Since PQ-MIM is compatible with generation, the conditional entropy model can be repurposed to predict the PQ codes for missing parts in an image. We show results for different dropping rates of transmitted tokens. PQ-MIM exhibits strong inpainting abilities. Even for the extreme case where half of the image patches are dropped, PQ-MIM can still retain a large percentage of the original image structure and details.</figcaption></figure>

redundancies among the patches, which corroborates our findings of its compression ability.

5.4.4 Limitations

Image compression, and compression of visual data in general, is an important technology
to scale the distribution of visual data. This is ever more important to cope with the growing
quantity of visual data that is streamed in the form of video and for augmented and virtual
reality applications. Compression is also critical to allow users with low-bandwidth connections
to benefit from applications relying on sharing of image, video, or virtual reality data.

**Caveats of learned neural compression models: biases and performance.** As with any machine learning model, potential biases in the training data may be transferred into the model via training. In our setting, this can affect the autoencoding reconstruction abilities for content under-represented in the training data, as well as the compression abilities of the model for such data. Such biases should be assessed before the deployment of the model. Beyond rate-distortion trade-offs, important evaluation dimensions include the energy and latency performance of compression models. Current neural compression methods, including ours, need to be further optimized to be competitive with existing codecs on these criteria.

**Specific limitations of our approach.** In our work, we specifically focus on a high compression regime, with compression rate lower than 0.6 bit per pixel. This is a favorable case for our approach: it benefits from the generative model capability inherited from the VQ-VAE. Compared to VQ, PQ allows for higher rates, but it becomes comparatively less effective when increasing the number of subquantizers, as one can deduce from the slopes of the rate-distortion curves in Fig. 5.5. It is also computationally more expensive.
5.5 Conclusion

In this chapter, we have revisited vector quantization for neural image compression. We introduced a product-quantization variants of VQ-VAE and shown that it has better scaling properties in terms of bit-rate. Additionally, we introduced a novel conditional entropy model based on masked image modeling. We have shown that combined with the quincunx partitioning pattern, PQ-MIM provides a strong reduction in bit-rate. PQ-MIM exhibits a competitive performance for PSNR and MS-SSIM metric compared to strong neural compression baselines. Furthermore, when we train PQ-MIM using perceptual losses, it provides a strong performance on multiple metrics (e.g. FID, KID and MS-SSIM) compared the strong baseline of HiFiC. Finally, we have shown that PQ-MIM can operate in a hybrid compression/generation mode where it can fill the gaps for non-transmitted patches.
<figure><img src="image_30.png" /><figcaption>Figure 5.11 – **Qualitative examples** More qualitative examples for image compression of high resolution images with PQ-MIM.</figcaption></figure>
# Chapter 6

## Learning a shared embedding space for six modalities with ImageBind

### Objectives

We present IMAGEBIND, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. IMAGEBIND can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications ‘out-of-the-box’ including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that IMAGEBIND serves as a new way to evaluate vision models for visual and non-visual tasks.

### Contents

<table><tr><td>6.1</td><td>Introduction</td><td>71</td></tr><tr><td>6.2</td><td>Related Work</td><td>72</td></tr><tr><td>6.3</td><td>Method</td><td>73</td></tr><tr><td>6.3.1</td><td>Preliminaries</td><td>73</td></tr><tr><td>6.3.2</td><td>Binding modalities with images</td><td>74</td></tr><tr><td>6.3.3</td><td>Implementation Details</td><td>75</td></tr><tr><td>6.4</td><td>Experiments</td><td>76</td></tr><tr><td>6.4.1</td><td>Emergent zero-shot classification</td><td>77</td></tr><tr><td>6.4.2</td><td>Comparison to prior work</td><td>78</td></tr><tr><td>6.4.3</td><td>Few-shot classification</td><td>79</td></tr><tr><td>6.4.4</td><td>Analysis and Applications</td><td>80</td></tr><tr><td>6.4.5</td><td>Pretraining details</td><td>82</td></tr><tr><td>6.4.6</td><td>Datasets and Metrics</td><td>82</td></tr><tr><td>6.5</td><td>Ablation Study</td><td>84</td></tr><tr><td>6.5.1</td><td>Scaling the Image Encoder</td><td>85</td></tr><tr><td>6.5.2</td><td>Training Loss and Architecture</td><td>85</td></tr><tr><td>6.6</td><td>Discussion and Limitations</td><td>88</td></tr></table>
<figure><img src="image_31.png" /><figcaption>Figure 6.1 – ImageBind’s joint embedding space enables novel multimodal capabilities. By aligning six modalities’ embedding into a common space, IMAGEBIND enables: 1) Cross-Modal Retrieval, which shows emergent alignment of modalities such as audio, depth or text, that aren’t observed together. 2) Adding embeddings from different modalities naturally composes their semantics. And 3) Audio-to-Image generation, by using our audio embeddings with a pre-trained DALLE-2 [Ramesh, 2022b] decoder designed to work with CLIP text embeddings.</figcaption></figure>

## 6.1 Introduction

A single image can bind together many experiences – an image of a beach can remind us of the sound of waves, the texture of the sand, a breeze, or even inspire a poem. This ‘binding’ property of images offers many sources of supervision to learn visual features, by aligning them with any of the sensory experiences associated with images. Ideally, for a single joint embedding space, visual features should be learned by aligning to all of these sensors. However, this requires acquiring all types and combinations of paired data with the same set of images, which is infeasible.

Recently, many methods learn image features aligned with text [Schuhmann, 2021; Radford, 2021; Mahajan, 2018; Yu, 2022; Yuan, 2021c; Jia, 2021a; Alayrac, 2022], audio [Morgado, 2021; Patrick, 2021; Owens, 2018; Bachmann, 2022; Tian, 2019; Arandjelovic, 2017] etc. These methods use a single pair of modalities or, at best, a few visual modalities. However, the final embeddings are limited to the pairs of modalities used for training. Thus, video-audio embeddings cannot directly be used for image-text tasks and vice versa. A major obstacle in learning true joint embedding is the absence of large quantities of multimodal data where all modalities are present together.

In this chapter, we present IMAGEBIND, which learns a single shared representation space by leveraging multiple types of image-paired data. It does not need datasets where all modalities co-occur with each other. Instead, we leverage the binding property of images and we show that just aligning each modality’s embedding to image embeddings leads to an emergent alignment across all of the modalities. In practice, IMAGEBIND leverages web-scale (image, text) paired data and combines it with naturally occurring paired data such as (video, audio), (image, depth) etc. to learn a single joint embedding space. This allows IMAGEBIND to implicitly align the text embeddings to other modalities such as audio, depth etc., enabling zero-shot recognition capabilities on that modality without explicit semantic or textual pairing. Moreover, we show
that it can be initialized with large-scale vision-language models such as CLIP [Radford, 2021], thereby leveraging the rich image and text representations of these models. Thus, IMAGEBIND can be applied to a variety of different modalities and tasks with little training.

We use large-scale image-text paired data along with naturally paired ‘self-supervised’ data across four new modalities - audio, depth, thermal, and Inertial Measurement Unit (IMU) readings – and show strong emergent zero-shot classification and retrieval performance on tasks for each of these modalities. These emergent properties improve as the underlying image representation is made stronger. On the audio classification and retrieval benchmarks, IMAGEBIND’s emergent zero-shot classification matches or outperforms specialist models trained with direct audio-text supervision on benchmarks like ESC, Clotho, AudioCaps. IMAGEBIND representations also outperform specialist supervised models on few-shot evaluation benchmarks. Finally, we show that IMAGEBIND’s joint embeddings can be used for a wide variety of compositional tasks as illustrated in fig. 6.1, including cross-modal retrieval, combining embeddings via arithmetic, detecting audio sources in images, and generating images given audio input.

## 6.2 Related Work

IMAGEBIND builds upon several advances in vision-language, multimodal, and self-supervised research.

**Language Image Pre-training.** Training images jointly with linguistic signals like words or sentences has been shown to be an effective method for zero-shot, open-vocabulary recognition and text to image retrieval [Frome, 2013; Faghri, 2017; Socher, 2014; Kiros, 2014]. Language as supervision can further be used for learning strong video representations [Alayrac, 2020; Miech, 2019b; Miech, 2020]. Joulin et al. [Joulin, 2016] show that using large-scale image dataset with noisy captions yields strong visual features. Recently, CLIP [Radford, 2021], ALIGN [Jia, 2021a] and Florence [Yuan, 2021c] collect large collections of image and text pairs and train models to embed image and language inputs in a joint space using contrastive learning, exhibiting impressive zero-shot performance. CoCa [Yu, 2022] adds an image captioning objective on top of the contrastive loss for improved performance. Flamingo [Alayrac, 2022] handles arbitrarily interleaved images and texts, and achieves state of the art on many few-shot learning benchmarks. LiT [Zhai, 2022] adopts contrastive training for fine-tuning and observes freezing image encoders works the best. This prior line of works mostly considers image and text, while our work enables zero-shot recognition on multiple modalities.

**Multi-Modal Learning.** Our work binds multiple modality representations in a joint embedding space. Prior works explored joint training of multiple modalities in a supervised [Girdhar, 2022b; Likhosherstov, 2021] or self-supervised contexts [Tian, 2019; Girdhar, 2022a; Wang, 2022; Morgado, 2021; Arandjelovic, 2017]. The success of image and language pre-training methods such as CLIP has inspired approaches that revisits learning deep semantic representations through matching other modalities with linguistic inputs. Various methods adapt CLIP to extract semantically strong video representations [Xue, 2022; Luo, 2021; Fang, 2021; Lin, 2022]. Most related to our method, Nagrani et al. [Nagrani, 2022] create a weakly-labeled dataset
for paired video-audio and captions that allows for training multi-modal video-audio encoder to match textual features resulting in strong audio and video retrieval and captioning performance. AudioCLIP [Guzhov, 2021] adds audio as an additional modality into a CLIP framework, enabling zero-shot audio classification. In contrast, IMAGEBIND does not require explicit paired data between all modalities and instead leverages image as a natural weak supervision for unifying modalities.

**Feature Alignment** Pre-trained CLIP models have been utilized as teachers to supervise other models due to the strength of its visual representations [Wei, 2022; Peng, 2022; Liu, 2022a]. Moreover, CLIP joint image and text embedding space has also been leveraged for a variety of zero-shot tasks like detection [Zhou, 2022; Gu, 2021], segmentation [Li, 2022], mesh animation [Youwang, 2022] etc. showing the power of joint embedding spaces. PointCLIP [Zhang, 2022] finds a pre-trained CLIP encoder can be used for 3D recognition by projecting a point cloud to a number of 2D depth map views, which in turn are encoded using CLIP visual encoder. In multilingual neural machine translation, a similar phenomenon to the emergence behavior of IMAGEBIND is commonly observed and utilized: if languages are trained in the same latent space through learned implicit bridging, translation can be done between language pairs on which no paired data is provided [Johnson, 2017; Lample, 2017].

## 6.3 Method

Our goal is to learn a single joint embedding space for all modalities by using images to bind them together. We align each modality’s embedding to image embeddings, such as text to image using web data and IMU to video using video data captured from egocentric cameras with IMU. We show that the resulting embedding space has a powerful emergent zero-shot behavior that automatically associates pairs of modalities without seeing any training data for that specific pair. We illustrate our approach in fig. 6.2.

### 6.3.1 Preliminaries

**Aligning specific pairs of modalities.** Contrastive learning [Hadsell, 2006] is a general technique for learning an embedding space by using pairs of related examples (positives) and unrelated examples (negatives). Using pairs of aligned observations, contrastive learning can align pairs of modalities such as (image, text) [Radford, 2021], (audio, text) [Guzhov, 2021], (image, depth) [Tian, 2019], (video, audio) [Morgado, 2021] etc. However, in each case, the joint embeddings are trained and evaluated using the same pairs of modalities. Thus, (video, audio) embeddings are not directly applicable for text-based tasks while (image, text) embeddings cannot be applied for audio tasks.

**Zero-shot image classification using text prompts.** CLIP [Radford, 2021] popularized a ‘zero-shot’ classification task based on an aligned (image, text) embedding space. This involves constructing a list of text descriptions that describe the classes in a dataset. An input image is classified based on its similarity to the text descriptions in the embedding space. Unlocking such zero-shot classification for other modalities requires specifically training using paired text data,
<figure><img src="image_32.png" /><figcaption>Figure 6.2 – ImageBind overview. Different modalities occur naturally aligned in different data sources, for instance images+text and video+audio in web data, depth or thermal information with images, IMU data in videos captured with egocentric cameras, etc. IMAGEBIND links all these modalities in a common embedding space, enabling new emergent alignments and capabilities.</figcaption></figure>

e.g., (audio, text) [Guzhov, 2021] or (point-clouds, text) [Zhang, 2022]. In contrast, IMAGEBIND unlocks zero-shot classification for modalities *without* paired text data.

### 6.3.2 Binding modalities with images

IMAGEBIND uses pairs of modalities ($\mathcal{I}$, $\mathcal{M}$), where $\mathcal{I}$ represents images and $\mathcal{M}$ is another modality, to learn a single joint embedding. We use large-scale web datasets with (image, text) pairings that span a wide range of semantic concepts. Additionally, we use the natural, self-supervised pairing of other modalities – audio, depth, thermal, and Intertial Measurement Unit (IMU) – with images.

Consider the pair of modalities ($\mathcal{I}$, $\mathcal{M}$) with aligned observations. Given an image $\mathbf{I}_i$ and its corresponding observation in the other modality $\mathbf{M}_i$, we encode them into normalized embeddings: $\mathbf{q}_i = f(\mathbf{I}_i)$ and $\mathbf{k}_i = g(\mathbf{M}_i)$ where $f, g$ are deep networks. The embeddings and the encoders are optimized using an InfoNCE [Oord, 2018] loss:

$$L_{\mathcal{I},\mathcal{M}} = - \log \frac{\exp(\mathbf{q}_i^\top \mathbf{k}_i / \tau)}{\exp(\mathbf{q}_i^\top \mathbf{k}_i / \tau) + \sum_{j \neq i} \exp(\mathbf{q}_j^\top \mathbf{k}_i / \tau)}, \quad (6.1)$$

where $\tau$ is a scalar temperature that controls the smoothness of the softmax distribution and $j$ denotes unrelated observations, also called ‘negatives’. We follow [Wu, 2018b] and consider every example $j \neq i$ in the mini-batch to be a negative. The loss makes the embeddings $\mathbf{q}_i$ and $\mathbf{k}_i$ closer in the joint embedding space, and thus aligns $\mathcal{I}$ and $\mathcal{M}$. In practice, we use a symmetric loss $L_{\mathcal{I},\mathcal{M}} + L_{\mathcal{M},\mathcal{I}}$.

**Emergent alignment of unseen pairs of modalities.** IMAGEBIND uses modalities paired with images, i.e., pairs of the form ($\mathcal{I}$, $\mathcal{M}$) to align each the embeddings from each modality $\mathcal{M}$ to those from images. We observe an emergent behavior in the embedding space that aligns two pairs of modalities ($\mathcal{M}_1, \mathcal{M}_2$) even though we only train using the pairs ($\mathcal{I}, \mathcal{M}_1$) and ($\mathcal{I}, \mathcal{M}_2$). This behavior allows us to perform a wide variety of zero-shot and cross-modal retrieval tasks without training for them. We achieve state-of-the-art zero-shot text-audio classification results without observing a single sample of paired (audio, text).
### 6.3.3 Implementation Details

IMAGEBIND is conceptually simple and can be implemented in many different ways. We deliberately choose a vanilla implementation that is flexible and allows for an effective study and easy adoption. In section 6.5, we present design decisions that are critical for good emergent 'binding'.

**Encoding modalities.** We use a Transformer architecture [Vaswani, 2017b] for all the modality encoders. We use the Vision Transformer (ViT) [Dosovitskiy, 2021a] for images. Following [Girdhar, 2022a], we use the same encoder for images and videos. We temporally inflate [Carreira, 2017] the patch projection layer of the ViT and use 2 frame video clips sampled from 2 seconds. We follow [Gong, 2021] for encoding audio and convert a 2 second audio sampled at 16kHz into spectrograms using 128 mel-spectrogram bins. As the spectrogram is also a 2D signal like an image, we use a ViT with a patch size of 16 and stride 10. We treat thermal images and depth images as one-channel images and also use a ViT to encode them. We follow [Girdhar, 2022b] to convert depth into disparity maps for scale invariance. We extract the IMU signal consisting of accelerometer and gyroscope measurements across the X, Y, and Z axes. We use 5 second clips resulting in 2K time step IMU readings which are projected using a 1D convolution with a kernel size of 8. The resulting sequence is encoded using a Transformer. Finally, we follow the text encoder design from CLIP [Radford, 2021].

We use separate encoders for images, text, audio, thermal images, depth images, and IMU. We add a modality-specific linear projection head on each encoder to obtain a fixed size *d* dimensional embedding, that is normalized and used in the InfoNCE loss from eq. (6.1). In addition to ease of learning, this setup allows us to also initialize a subset of the encoders using pretrained models, e.g., the image and text encoder using CLIP [Radford, 2021] or OpenCLIP [Ilharco, 2021].

#### 6.3.3.1 Inference implementation details

**Audio/Video:** For both these temporal modalities (whether operated upon together during pre-training or separately during inference), we sample fixed length clips to operate on. During training, we randomly sample a clip, typically 2s in length. At inference time, we uniformly sample multiple clips to cover the full length of the input sample. For instance, for 5s ESC videos, we would sample ⌈5/2⌋ = 3 clips. For video clips, we sample a fixed number of frames from each clip. For audio, we process each raw audio waveform by sampling it at 16KHz followed by extracting a log mel spectrogram with 128 frequency bins using a 25ms Hamming window with hop length of 10ms. Hence, for a *t* second audio we get a 128 × 100*t* dimensional input.

**IMU:** For IMU, we sample fixed length clips of 5 seconds, centered around time-stamps that are aligned with narrations. For each clip, we get a 6 × 2000 dimensional input and we measure the zero-shot performance for scenario classification using each clip as an independent testing sample.
Chapter 6. Learning a shared embedding space for six modalities with ImageBind

<table><caption>Table 6.1 – **Emergent zero-shot classification datasets** for audio, depth, thermal, and Inertial Measurement Unit (IMU) modalities. We evaluate IMAGEBIND without training for any of these tasks and without training on paired text data for these modalities. For each dataset, we report the task (classification or retrieval), number of classes (#cls), metric for evaluation (Accuracy or mean Average Precision), and the number of test samples (#test).</caption><thead><tr><th>Dataset</th><th>Task</th><th>#cls</th><th>Metric</th><th>#test</th></tr></thead><tbody><tr><td>Audioset Audio-only (AS-A) [Gemmeke, 2017]</td><td>Audio cls.</td><td>527</td><td>mAP</td><td>19048</td></tr><tr><td>ESC 5-folds (ESC) [Piczak, 2015]</td><td>Audio cls.</td><td>50</td><td>Acc</td><td>400</td></tr><tr><td>Clotho (Clotho) [Font, 2013]</td><td>Retrieval</td><td>-</td><td>Recall</td><td>1045</td></tr><tr><td>AudioCaps (AudioCaps) [Kim, 2019]</td><td>Retrieval</td><td>-</td><td>Recall</td><td>796</td></tr><tr><td>VGGSound (VGGS) [Chen, 2020a]</td><td>Audio cls.</td><td>309</td><td>Acc</td><td>14073</td></tr><tr><td>SUN Depth-only (SUN-D) [Song, 2015]</td><td>Scene cls.</td><td>19</td><td>Acc</td><td>4660</td></tr><tr><td>NYU-v2 Depth-only (NYU-D) [Silberman, 2012]</td><td>Scene cls.</td><td>10</td><td>Acc</td><td>653</td></tr><tr><td>LLVIP (LLVIP) [Jia, 2021b]</td><td>Person cls.</td><td>2</td><td>Acc</td><td>15809</td></tr><tr><td>Ego4D (Ego4D) [Grauman, 2022]</td><td>Scenario cls.</td><td>108</td><td>Acc</td><td>68865</td></tr></tbody></table>

<table><caption>Table 6.2 – **Emergent zero-shot classification** of IMAGEBIND using text prompts highlighted in blue. IMAGEBIND aligns images with text, depth, audio, thermal and IMU modalities. The resulting embedding space can associate text embeddings with the non-image modalities, and leads to strong emergent zero-shot classification. We show strong performance even on non-visual modalities such as audio and IMU. We compare to ‘Text Paired’ baselines wherever possible, which trains with paired text data for that modality. *We use the OpenCLIP ViT-H [Ilharco, 2021] on depth rendered as grayscale images. †[Guzhov, 2021] that uses AS class names as supervision during training, and hence is not “zero-shot”. Overall, IMAGEBIND shows strong emergent zero-shot performance, even compared to such upper bounds. We also report the absolute state-of-the-art (SOTA) on each dataset for reference, which typically uses additional supervision, model ensembles etc. We report the top-1 classification accuracy for all datasets except MSR-VTT (Recall@1) and Audioset Audio-only (mAP).</caption><thead><tr><th></th><th>IN1K</th><th>P365</th><th>K400</th><th>MSR-VTT</th><th>NYU-D</th><th>SUN-D</th><th>AS-A</th><th>VGGS</th><th>ESC</th><th>LLVIP</th><th>▲Ego4D</th></tr></thead><tbody><tr><td>Random</td><td>0.1</td><td>0.27</td><td>0.25</td><td>0.1</td><td>10.0</td><td>5.26</td><td>0.62</td><td>0.32</td><td>2.75</td><td>50.0</td><td>0.9</td></tr><tr><td>IMAGEBIND</td><td>77.7</td><td>45.4</td><td>50.0</td><td>36.1</td><td>54.0</td><td>35.1</td><td>17.6</td><td>27.8</td><td>66.9</td><td>63.4</td><td>25.0</td></tr><tr><td>Text Paired</td><td>-</td><td>-</td><td>-</td><td>-</td><td>41.9*</td><td>25.4*</td><td>28.4†</td><td>-</td><td>68.6†</td><td>-</td><td>-</td></tr><tr><td>Absolute SOTA</td><td>91.0</td><td>60.7</td><td>89.9</td><td>57.7</td><td>76.7</td><td>64.9</td><td>49.6</td><td>52.5</td><td>97.0</td><td>-</td><td>-</td></tr></tbody></table>

## 6.4 Experiments

**Naturally paired modalities and datasets.** We use IMAGEBIND on six modalities - image/video, text, audio, depth, thermal images, and IMU. As described in section 6.3.3, we treat videos as 2 frame images and process them the same as images. For the naturally available paired data, we use the (video, audio) pairs from the Audioset dataset [Gemmeke, 2017], (image, depth) pairs from the SUN RGB-D dataset [Song, 2015], (image, thermal) pairs from the LLVIP dataset [Jia, 2021b] and (video, IMU) pairs from the Ego4D dataset [Grauman, 2022]. For these pairs of modalities, we do not use any extra supervision like class labels, text etc. Since SUN RGB-D and LLVIP are relatively small, we follow [Girdhar, 2022b] and replicate them 50× for training.

**Large scale image-text pairs.** We leverage image-text supervision from large-scale web data [Radford, 2021]. For ease of experimentation, we use pretrained models that are trained on billions of (image, text) pairs. Specifically, we use the pretrained vision (ViT-H 630M params) and text encoders (302M params) from OpenCLIP [Ilharco, 2021] in our experiments.
**Encoders for each modality.** We convert audio into 2D mel-spectrograms [Gong, 2021], and thermal and depth modalities into 1 channel images and use ViT-B, ViT-S encoders respectively. The image and text encoders are kept frozen during the IMAGEBIND training and the audio, depth, thermal, and IMU encoders are updated.

**Emergent zero-shot vs. zero-shot.** Methods such as CLIP [Radford, 2021], AudioCLIP [Guzhov, 2021] etc. train with modality pairs, (image, text) and (audio, text), to demonstrate zero-shot classification using text-prompts for the same modality. In contrast, IMAGEBIND binds modalities together using only image-paired data. Thus, just by training on (image, text) and (image, audio), IMAGEBIND can perform zero-shot classification of audio using text prompts. As we do not directly train for this ability, we term it *emergent zero-shot classification* to distinguish it from methods that specifically train using paired text-supervision for all modalities.

**Evaluation on downstream tasks.** We comprehensively evaluate IMAGEBIND on a many different downstream tasks using different protocols. We summarize the main datasets used for evaluation in table 6.1.

### 6.4.1 Emergent zero-shot classification

We evaluate IMAGEBIND on emergent zero-shot classification and use the text prompt templates from [Radford, 2021]. We report the results in table 6.2. Each task measures IMAGEBIND's ability to associate text embeddings to the other modalities without observing them together during training. Given the novelty of our problem setting, there are no "fair" baselines to compare IMAGEBIND with. Nevertheless, we compare to prior work that uses text paired with certain modalities (e.g. audio [Nagrani, 2022; Guzhov, 2021]), and for certain "visual-like" modalities such as depth and thermal, we use the CLIP model directly. We also report the best reported supervised upper bound per benchmark.

IMAGEBIND achieves a high emergent zero-shot classification performance. On each benchmark, IMAGEBIND achieves strong gains and even compares favorably to supervised specialist models trained for the specific modality and task. These results demonstrate that IMAGEBIND aligns the modalities and implicitly transfers the text supervision associated with images to other modalities like audio. In particular, IMAGEBIND shows strong alignment for non-visual modalities like audio and IMU suggesting that their naturally available pairing with images is a powerful source of supervision. For completeness, we also report the standard zero-shot image (ImageNet [Russakovsky, 2015] - IN1K, Places-365 [Zhou, 2014] - P365) and video (Kinetics400 [Kay, 2017] - K400, MSR-VTT 1k-A [Xu, 2016] - MSR-VTT) tasks. As the image & text encoders are initialized (and frozen) using OpenCLIP, these results match those of OpenCLIP.

#### 6.4.1.1 Zero-shot evaluation details

**Query Templates.** For all evaluations, we use the default set of templates from CLIP [Radford, 2021].<fnref n="1" /> Note that we use the same templates for non visual modalities like audio and depth as

<footnote n="1">1. https://github.com/openai/CLIP/blob/main/notebooks/Prompt_Engineering_for_ImageNet.ipynb</footnote>
Chapter 6. Learning a shared embedding space for six modalities with ImageBind

<table><caption>Table 6.3 – **Emergent zero-shot audio retrieval and classification.** We compare IMAGEBIND to prior work on zero-shot audio retrieval and audio classification. Without using audio-specific supervision, IMAGEBIND outperforms prior methods on zero-shot retrieval and has comparable performance on the classification task. IMAGEBIND’s emergent zero-shot performance approaches those of specialist supervised models.</caption><thead><tr><th></th><th>Emergent</th><th colspan="2">Clotho</th><th colspan="2">AudioCaps</th><th>ESC</th></tr><tr><th></th><th></th><th>R@1</th><th>R@10</th><th>R@1</th><th>R@10</th><th>Top-1</th></tr></thead><tbody><tr><td><strong>Uses audio and text supervision</strong><br>AudioCLIP [Guzhov, 2021]</td><td>✗</td><td>—</td><td>—</td><td>—</td><td>—</td><td><strong>68.6</strong></td></tr><tr><td><strong>Uses audio and text loss</strong><br>AVFIC [Nagrani, 2022]</td><td>✗</td><td>3.0</td><td>17.5</td><td>8.7</td><td>37.7</td><td>—</td></tr><tr><td><strong>No audio and text supervision</strong><br>IMAGEBIND</td><td>✓</td><td><strong>6.0</strong></td><td><strong>28.4</strong></td><td><strong>9.3</strong></td><td><strong>42.3</strong></td><td>66.9</td></tr><tr><td><em>Supervised</em><br>AVFIC finetuned [Nagrani, 2022]</td><td>✗</td><td>8.4</td><td>38.6</td><td>—</td><td>—</td><td>—</td></tr><tr><td>ARNLQ [Oncescu, 2021]</td><td>✗</td><td>12.6</td><td>45.4</td><td>24.3</td><td>72.1</td><td>—</td></tr></tbody></table>

<table><caption>Table 6.4 – **Zero-shot text based retrieval** on MSR-VTT 1K-A. We compare IMAGEBIND’s emergent retrieval performance using audio and observe that it performs favorably to methods that use the stronger video modality for retrieval.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Modality</th><th rowspan="2">Emergent</th><th colspan="3">MSR-VTT</th></tr><tr><th>R@1</th><th>R@5</th><th>R@10</th></tr></thead><tbody><tr><td>MIL-NCE [Miech, 2019a]</td><td>V</td><td>✗</td><td>8.6</td><td>16.9</td><td>25.8</td></tr><tr><td>SupportSet [Patrick, 2020]</td><td>V</td><td>✗</td><td>10.4</td><td>22.2</td><td>30.0</td></tr><tr><td>FIT [Bain, 2021]</td><td>V</td><td>✗</td><td>15.4</td><td>33.6</td><td>44.1</td></tr><tr><td>AVFIC [Nagrani, 2022]</td><td>A+V</td><td>✗</td><td>19.4</td><td>39.5</td><td>50.3</td></tr><tr><td>IMAGEBIND</td><td>A</td><td>✓</td><td>6.8</td><td>18.5</td><td>27.2</td></tr><tr><td>IMAGEBIND</td><td>A+V</td><td>✗</td><td>36.8</td><td>61.8</td><td>70.0</td></tr></tbody></table>

well since we only use semantic/textual supervision associated with images.

### 6.4.2 Comparison to prior work

We now compare IMAGEBIND against prior work in zero-shot retrieval and classification tasks.

**Zero-shot text to audio retrieval and classification.** Unlike IMAGEBIND, prior work trains using paired data for that modality, e.g., AudioCLIP [Guzhov, 2021] uses (audio, text) supervision and AVFIC [Nagrani, 2021] uses automatically mined (audio, text) pairs. We compare their zero-shot text to audio retrieval and classification performance to IMAGEBIND’s emergent retrieval and classification in table 6.3.

IMAGEBIND significantly outperforms prior work on the audio text retrieval benchmarks. On the Clotho dataset, IMAGEBIND has double the performance of AVFIC despite not using any text pairing for audio during training. Compared to the supervised AudioCLIP model, IMAGEBIND achieves comparable audio classification performance on ESC. Note that AudioCLIP uses class names from AudioSet as text targets for audio-text training, hence is referred to as ‘supervised’.
IMAGEBIND's strong performance on all three benchmarks validates its ability to align the audio and text modalities using images as a bridge.

**Text to audio and video retrieval.** We use the MSR-VTT 1k-A benchmark to evaluate the text to audio and video retrieval performance in table. 6.4. Only using audio, IMAGEBIND achieves strong emergent retrieval performance compared to the video retrieval performance of prior work like MIL-NCE. The text to video performance for our model is strong (36.1% R@1 in table. 6.2) as it uses OpenCLIP's vision and text encoders and outperforms many prior methods. However, combining the audio and video modalities further boosts performance showing the utility of IMAGEBIND's features over an already strong retrieval model.

### 6.4.3 Few-shot classification

<figure><img src="image_33.png" /><figcaption>Figure 6.3 – Few-shot classification on audio and depth. We report the emergent zero-shot classification performance on each benchmark (denoted by ⋆). We train linear classifiers on fixed features for the ≥ 1-shot case. (Left) In all settings, IMAGEBIND outperforms the self-supervised AudioMAE model. IMAGEBIND even outperforms a supervised AudioMAE model upto 4 shot learning showing its strong generalization. (Right) We compare with the MultiMAE model trained with images, depth, and semantic segmentation masks. IMAGEBIND outperforms MultiMAE across all few-shot settings on few-shot depth classification.</figcaption></figure>

We now evaluate the label-efficiency of IMAGEBIND by evaluating on few-shot classification. We use the audio and depth encoders from IMAGEBIND and evaluate them on audio and depth classification respectively in fig. 6.3. For ≥ 1-shot results, we follow [Radford, 2021; Morgado, 2021] and train linear classifiers on fixed features.

On few-shot audio classification (fig. 6.3 left), we compare with (1) self-supervised AudioMAE model trained on audio from Audioset and (2) a supervised AudioMAE model finetuned on audio classification. Both baselines use the same capacity ViT-B audio encoder as IMAGEBIND. IMAGEBIND significantly outperforms the AudioMAE model on all settings with gains of ~40% accuracy in top-1 accuracy on ≤ 4-shot classification. IMAGEBIND also matches or outperforms the supervised model on ≥ 1-shot classification. IMAGEBIND's emergent zero-shot performance surpasses the supervised ≤ 2-shot performance.

For few-shot depth classification, we compare with the multimodal MultiMAE [Bachmann,
Chapter 6. Learning a shared embedding space for six modalities with ImageBind

<figure><img src="image_34.png" /><figcaption>Figure 6.4 – **Embedding space arithmetic** where we add image and audio embeddings, and use them for image retrieval. The composed embeddings naturally capture semantics from different modalities. Embeddings from an image of fruits + the sound of birds retrieves images of birds surrounded by fruits.</figcaption></figure>

[2022] ViT-B/16 model trained on images, depth, and semantic segmentation data. IMAGEBIND significantly outperforms MultiMAE across all the few-shot settings. Altogether, these results show the strong generalization of IMAGEBIND audio and depth features trained with image alignment.

6.4.3.1 Few-shot evaluation details

For the few-shot results in Figures 6.3 using the ESC and SUN datasets, we sampled *k* training samples per class, where *k* ∈ {1, 2, 4, 8}. We fix the *k* samples such that our model and the baselines use exactly the same samples during training. For all few-shot evaluations, including the baselines, we freeze the encoder parameters and only train a linear classifier.

**Audio:** For audio few-shot training with ESC, our model and the baselines are trained using AdamW with a learning rate of 1.6 × 10⁻³ and weight decay of 0.05 for 50 epochs.

**Depth:** For depth few-shot training with SUN, our model and the baselines are trained using AdamW with a learning rate of 10⁻² and no weight decay for 60 epochs.

6.4.4 Analysis and Applications

**Multimodal embedding space arithmetic.** We study whether IMAGEBIND's embeddings can be used to compose information across modalities. In fig. 6.4, we show image retrievals obtained by adding together image and audio embeddings. The joint embedding space allows for us to compose two embeddings: e.g., image of fruits on a table + sound of chirping birds and retrieve an image that contains both these concepts, i.e., fruits on trees with birds. Such
<figure><img src="image_35.png" /><figcaption>Figure 6.5 – **Object detection with audio queries.** Simply replacing Detic [Zhou, 2022]'s CLIP-based 'class' embeddings with our audio embeddings leads to an object detector promptable with audio. This requires no re-training of any model.</figcaption></figure>

*emergent compositionality* whereby semantic content from different modalities can be composed
will likely enable a rich variety of compositional tasks.

Without re-training, we can ‘upgrade’ existing vision models that use CLIP embeddings to
use IMAGEBIND embeddings from other modalities such as audio.

**Upgrading text-based detectors to audio-based.** We use a pretrained text-based detection model, Detic [Zhou, 2022], and simply replace its CLIP-based ‘class’ (text) embeddings with IMAGEBIND’s audio embeddings. Without training, this creates an ‘audio’-based detector that can detect and segment objects based on audio prompts. As shown in fig. 6.5, we can prompt the detector with the barking sound of a dog to localize a dog.

**Upgrading text-based diffusion models to audio-based.** We use a pretrained DALLE-2 [Ramesh, 2022b] diffusion model (private reimplementation) and replace its prompt embeddings by our audio embeddings. In fig. 6.1, we observe that we can repurpose the diffusion model to generate plausible images using different types of sounds.

6.4.4.1 Qualitative evaluation details

**Cross-modal nearest neighbors.** We perform the retrieval on the embedding feature after temperature scaling. The nearest neighbors are computed using cosine distance. In fig. 6.1, we show retrievals for audio from ESC, image retrievals from IN1K and COCO, depth from SUN-D, and text from AudioCaps.

**Embedding arithmetic.** For arithmetic, we again use the embedding features after tempera-
ture scaling. We l₂ normalize the features and sum the embeddings after scaling them by 0.5.
We use the combined feature to perform nearest neighbor retrieval using cosine distance, as
described above. In fig. 6.1, we show combination of images and audio from IN1K and ESC,
and show retrievals from IN1K.

**Audio→Image Generation.** For generating images from audio clips, we rely on an in-house
reproduced implementation of DALLE-2 [Ramesh, 2022b]. In DALLE-2, to produce images from
Chapter 6. Learning a shared embedding space for six modalities with ImageBind

<table><caption>Table 6.5 – Pretraining hyperparameters</caption><thead><tr><th>Config</th><th>AS</th><th>SUN</th><th>LLVIP</th><th>Ego4D</th></tr></thead><tbody><tr><td>Vision encoder</td><td></td><td colspan="3">ViT-Huge</td></tr><tr><td>embedding dim.</td><td>768</td><td>384</td><td>768</td><td>512</td></tr><tr><td>number of heads</td><td>12</td><td>8</td><td>12</td><td>8</td></tr><tr><td>number of layers</td><td>12</td><td>12</td><td>12</td><td>6</td></tr><tr><td>Optimizer</td><td></td><td colspan="3">AdamW</td></tr><tr><td>Optimizer Momentum</td><td></td><td colspan="3">β₁ = 0.9, β₂ = 0.95</td></tr><tr><td>Peak learning rate</td><td>1.6e-3</td><td>1.6e-3</td><td>5e-4</td><td>5e-4</td></tr><tr><td>Weight decay</td><td>0.2</td><td>0.2</td><td>0.05</td><td>0.5</td></tr><tr><td>Batch size</td><td>2048</td><td>512</td><td>512</td><td>512</td></tr><tr><td>Gradient clipping</td><td>1.0</td><td>1.0</td><td>5.0</td><td>1.0</td></tr><tr><td>Warmup epochs</td><td></td><td colspan="3">2</td></tr><tr><td>Sample replication</td><td>1.25</td><td>50</td><td>25</td><td>1.0</td></tr><tr><td>Total epochs</td><td>64</td><td>64</td><td>64</td><td>8</td></tr><tr><td>Stoch. Depth [Huang, 2016]</td><td>0.1</td><td>0.0</td><td>0.0</td><td>0.7</td></tr><tr><td>Temperature</td><td>0.05</td><td>0.2</td><td>0.1</td><td>0.2</td></tr><tr><td>Augmentations:</td><td></td><td></td><td></td><td></td></tr><tr><td>RandomResizedCrop</td><td></td><td colspan="3">224px</td></tr><tr><td>size</td><td>—</td><td>Bilinear</td><td>Bilinear</td><td>—</td></tr><tr><td>interpolation</td><td>—</td><td>p = 0.5</td><td>p = 0.5</td><td>—</td></tr><tr><td>RandomHorizontalFlip</td><td>—</td><td>p = 0.25</td><td>p = 0.25</td><td>—</td></tr><tr><td>RandomErase</td><td>—</td><td>9/0.5</td><td>9/0.5</td><td>—</td></tr><tr><td>RandAugment</td><td>—</td><td>0.4</td><td>0.4</td><td>—</td></tr><tr><td>Color Jitter</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Frequency masking</td><td>12</td><td>—</td><td>—</td><td>—</td></tr></tbody></table>

text prompts, the image generation model relies on text embeddings produced by the pre-trained CLIP-L/14 text encoder. Since IMAGEBIND naturally aligns CLIP's-embedding space to that of other modalities proposed in the chapter, we can upgrade the DALLE-2 model to generate images by prompting it with these new unseen modalities. We achieve zero-shot audio to image generation with DALLE-2 by simply using the temperature-scaled audio embeddings generated by IMAGEBIND's audio encoder as a proxy for the CLIP's text embeddings in the DALLE-2's image generation model.

**Detecting objects using audio.** We extract all audio descriptors from the validation set of ESC using an IMAGEBIND ViT-B/32 encoder, yielding 400 descriptors in total. We use an off-the-shelf CLIP-based Detic [Zhou, 2022] model and use the audio descriptors as the classifier for Detic in place of CLIP text-based ‘class’ embeddings. We use a score threshold of 0.9 for the qualitative results in Figure 6.5.

### 6.4.5 Pretraining details

In Table 6.5 we detail the hyperparameters used to pre-train each of the models reported in Table 6.2.

### 6.4.6 Datasets and Metrics

Audioset (AS) [Gemmeke, 2017]. This dataset is used for both training and evaluation. It contains 10s videos from YouTube annotated into 527 classes. It consists of 3 pre-defined splits,
the balanced split with about 20K videos, test split with 18K videos, and an unbalanced training split with about 2M videos. For **training**, we use the 2M unbalanced set without any labels, and only use it for audio-video matching. For **zero-shot evaluation** in table 6.2, we use the test set and compute logits for each class using the textual class names along with the templates as described later in section 6.4.1.1. The metric used is top-1 accuracy.

**ESC-50 (ESC)** [Piczak, 2015]. We use this dataset for evaluating the learned representations in a zero-shot manner. The task here is “Environmental Sound Classification” (ESC). It consists of 2000 5s audio clips classified into 50 classes. It has pre-defined 5 fold evaluation, each consisting of 400 test audio clips. In this work, we compute 0-shot predictions on the evaluation set for each fold and report the 5-fold average performance. For ablations we use only the first fold for computational ease. The metric used is top-1 accuracy.

**Clotho (Clotho)** [Font, 2013]. This is a dataset of audio from the Freesound platform with textual descriptions. It consists of a dev and test set of 2893 and 1045 audio clips respectively, with each clip associated with 5 descriptions. We consider the text→audio retrieval task, and consider each of the 5 associated captions as a separate test query and retrieve from the set of audio clips. The metric used is recall@K, where a given test query is assumed to be correctly solved if the ground truth audio is retrieved within the top-K retrieved audio clips.

**AudioCaps (AudioCaps)** [Kim, 2019]. This is a dataset of audiovisual clips from YouTube accompanied by textual descriptions. It consists of clips from the Audioset dataset as described earlier. We use the splits as provided in [Oncescu, 2021],<fnref n="2" /> which removes clips that overlap with the VGGSound dataset. We end up with 48198 training, 418 validation and 796 test clips. We only use the test set for zero-shot evaluation of our model. The task is text→audio retrieval, and evaluation is performed using recall@K.

**VGGSound (VGGS)** [Chen, 2020a]. This dataset contains about 200K video clips of 10s length, annotated with 309 sound classes consisting of human actions, sound-emitting objects and human-object interactions. We only use the audio from the test set (with 14073 clips) for 0-shot classification. The evaluation is done using the top-1 accuracy metric.

**SUN RGB-D (SUN).** We use the registered RGB and Depth maps provided in the SUN RGB-D [Song, 2015] dataset train set (~5K pairs) for training our model. We follow [Girdhar, 2022b] to post process the depth maps in two steps - 1) we use in-filled depth values and 2) convert them to disparity for scale normalization. This dataset is only used in training, so we do not use any metadata or class labels.

**SUN Depth-only (SUN-D).** We use only the ~5K depth maps from the val split of the SUN RGB-D [Song, 2015] dataset and denote them as SUN Depth-only. This dataset is only used for evaluation and we do not use the RGB images. We process the depth maps similar to SUN RGB-D (in-filled depth, converted to disparity). We use the 19 scene classes in the dataset and use their class names for constructing the zero-shot classification templates.

**NYU-v2 Depth-only (NYU-D).** We use the 794 val set depth maps from the NYU-v2 Depth-only [Silberman, 2012] dataset for evaluation only. We post-process the depth similar to

<footnote n="2">2. https://www.robots.ox.ac.uk/~vgg/research/audio-retrieval/resources/benchmark-files/AudioCaps_retrieval_dataset.tar.gz</footnote>
SUN Depth-only. We use the 10 scene class names in the dataset. The 10th scene class, called 'other', correspond to 18 different semantic classes – ['basement', 'cafe', 'computer lab', 'conference room', 'dinette', 'exercise room', 'foyer', 'furniture store', 'home storage', 'indoor balcony', 'laundry room', 'office kitchen', 'playroom', 'printer room', 'reception room', 'student lounge', 'study', 'study room)]. For zero-shot evaluation, we compute the cosine similarity of the 10th class as the maximum cosine similarity among these 18 classnames.

**LLVIP (LLVIP).** The LLVIP dataset [Jia, 2021b] consists of RGB image and Thermal (infrared low-light) image pairs. The dataset was collected in an outdoor setting using fixed cameras observing street scenes and contains RGB images taken in a low-light paired with infrared images (8~14um frequency). The RGB thermal pairs are registered in the dataset release. For training, we use the **train** set with 12025 RGB image and thermal pairs. For evaluation, we use the **val** set with 3463 pairs of RGB and thermal images. Since the original dataset is designed for detection, we post process it for a binary classification task. We crop out pedestrian bounding boxes and random bounding boxes (same aspect ratio and size as pedestrian) to create a balanced set of 15809 total boxes (7931 'person' boxes). For zero-shot classification, we use the following class names for the 'person' class - ['person', 'man', 'woman', 'people'], and ['street', 'road', 'car', 'light', 'tree'] for the background class.

**Ego4D (Ego4D)** [Grauman, 2022]. For the Ego4D dataset, we consider the task of scenario classification. There are 108 unique scenarios present in the 9,645 videos of the Ego4D dataset. We filter out all videos annotated with more than one scenario which yields 7,485 videos with a single scenario assigned. For each video, We select all time-stamps that contains a synchronized IMU signal as well as aligned narrations. We sample 5 second clips around each time-stamp. The dataset is split randomly such that we have 510,142 clips for training, and 68,865 clips for testing. During training we only use the video frames and their corresponding IMU signal. We use the test split to measure zero-shot scenario classification performance, where each clip of IMU signal is assigned the video-level scenario label as its ground-truth.

## 6.5 Ablation Study

We investigate various design choices for learning a joint embedding space for different modalities. Since the ablation experimental setup is similar to section 6.4, we only note the main differences (full details in section 6.4.5). We report results on the ESC fold-1 for the ablation study. We use a ViT-B encoder for the image, audio, depth, and thermal modalities by default and train them for 16 epochs (vs. 32 epochs in section 6.4). For IMU we use a lightweight 6 layer encoder with 512 dimensional width and 8 heads, and train it for 8 epochs. The text encoder follows [Radford, 2021] and is a twelve layer Transformer with a width of 512 dimensions. We initialize the image and text encoder from the CLIP model [Radford, 2021].
<figure><img src="image_36.png" /><figcaption>Figure 6.6 – Scaling the image encoder size while keeping the other modality encoders’ size fixed. We measure the performance on the emergent zero-shot classification of depth, audio, thermal, and IMU modalities. Scaling the image encoder significantly improves the zero-shot classification results suggesting that a stronger visual representation improves the ‘binding’ of modalities.</figcaption></figure>

### 6.5.1 Scaling the Image Encoder

The central idea in IMAGEBIND is aligning the embeddings of all modalities to image embeddings. Thus, the image embeddings plays a central role in the emergent alignment of unseen modalities and we study their effect on the emergent zero-shot performance. We vary the size of the image encoder and train an encoder for the depth, audio *etc.* modalities to match the image representation. To isolate the effect of the image representation, we fix the size of the other modality encoders. We use the pretrained CLIP (ViT-B and ViT-L) and OpenCLIP (ViT-H) image and text encoders for this experiment. Our results in fig. 6.6 show that IMAGEBIND’s emergent zero-shot performance on all modalities improves with better visual features. For depth and audio classification, the stronger ViT-H *vs.* the ViT-B image encoder, provides a gain of 7% and 4% respectively. Thus, stronger visual features can improve recognition performance even on non-visual modalities.

### 6.5.2 Training Loss and Architecture

We study the effect of the training design choices on the emergent zero-shot classification. We focus on two modalities with different characteristics - depth which is visual and spatial, and audio which is non-visual and has a temporal component. We found that studying these diverse modalities led to robust and transferable design decisions.

**Contrastive loss temperature.** We study the effect of the temperature $\tau$ (eq. (6.1)) in table 6.6a. We experiment with a learnable temperature initialized to 0.07 (parametrized in the log-scale) following [Radford, 2021] *vs.* various values of fixed temperatures. Unlike [Radford, 2021], we observe that a fixed temperature is best for depth, audio and IMU classification. Ad-
Chapter 6. Learning a shared embedding space for six modalities with ImageBind

<table><thead><tr><th>Temp →</th><th>Learn</th><th>0.05</th><th>0.07</th><th>0.2</th><th>1.0</th></tr></thead><tbody><tr><td>SUN-D</td><td>24.1</td><td>27.0</td><td>27.3</td><td>26.7</td><td>28.0</td></tr><tr><td>ESC</td><td>54.8</td><td>56.7</td><td>52.4</td><td>45.4</td><td>24.3</td></tr></tbody></table>

(a) **Temperature for loss.**

<table><thead><tr><th>Epochs →</th><th>16</th><th>32</th><th>64</th></tr></thead><tbody><tr><td>SUN-D</td><td>26.7</td><td>27.9</td><td>29.9</td></tr><tr><td>ESC</td><td>56.7</td><td>61.3</td><td>62.9</td></tr></tbody></table>

(d) **Training epochs.**

<table><thead><tr><th>Data aug →</th><th>None</th><th>RandErase</th></tr></thead><tbody><tr><td>SUN-D</td><td>24.2</td><td>26.7</td></tr></tbody></table>

(g) **Depth data aug.**

<table><thead><tr><th>Proj head →</th><th>Linear</th><th>MLP</th></tr></thead><tbody><tr><td>SUN-D</td><td>26.7</td><td>26.5</td></tr><tr><td>ESC</td><td>56.7</td><td>51.0</td></tr></tbody></table>

(b) **Projection Head.**

<table><thead><tr><th>Data aug →</th><th>Basic</th><th>Strong</th></tr></thead><tbody><tr><td>SUN-D</td><td>25.4</td><td>26.7</td></tr><tr><td>ESC</td><td>56.7</td><td>22.6</td></tr></tbody></table>

(e) **Data aug for image.**

<table><thead><tr><th>Temporal align→</th><th>None</th><th>Aligned</th></tr></thead><tbody><tr><td>ESC</td><td>55.7</td><td>56.7</td></tr></tbody></table>

(h) **Temporal alignment of audio.**

<table><thead><tr><th>Batch size →</th><th>512</th><th>1k</th><th>2k</th><th>4k</th></tr></thead><tbody><tr><td>NYU-D</td><td>47.3</td><td>46.5</td><td>43.0</td><td>39.9</td></tr><tr><td>ESC</td><td>39.4</td><td>53.9</td><td>56.7</td><td>53.9</td></tr></tbody></table>

(c) **Batch size.**

<table><thead><tr><th>Spatial align →</th><th>None</th><th>Aligned</th></tr></thead><tbody><tr><td>SUN-D</td><td>16.0</td><td>26.7</td></tr></tbody></table>

(f) **Spatial alignment of depth.**

<table><thead><tr><th>Data aug →</th><th>Basic</th><th>+Freq mask</th></tr></thead><tbody><tr><td>ESC</td><td>56.5</td><td>56.7</td></tr></tbody></table>

(i) **Audio data aug.**

Table 6.6 – **Training loss and architecture** design decisions and their impact on emergent zero-shot classification. Settings for results in section 6.4 highlighted in gray. (a) A fixed temperature in the contrastive loss outperforms a learnable one for all modalities. (b) A linear projection head for computing the depth or audio embedding works better than an MLP head. (c) A smaller batch size works better for depth classification presumably because of the smaller size of (image, depth) datasets. (d) Longer training improves the zero-shot classification performance for both modalities. (e) Stronger image augmentation improves depth classification while basic augmentation significantly improves audio classification. (f, g) Using spatially aligned image and depth crops when training IMAGEBIND significantly improves performance. Similarly, RandErase augmentation is critical to good zero-shot classification on depth. (h, i) Temporally aligned audio and video matching gives improved performance and using frequency augmentation for audio gives a slight improvement.

ditionally, we see that a higher temperature is better for training the depth, thermal, and IMU encoders, whereas a lower temperature works best for the audio modality.

**Projection head.** We vary the projection head used for each encoder from a linear layer to an MLP with 768 hidden dimensions. The results in table. 6.6b show that a linear projection performs better for both modalities. This is in contrast to standard self-supervised methods like SimCLR [Chen, 2020c] whose performance improves with MLP projection heads.

**Training epochs.** We vary the number training epochs and report the classification performance in table. 6.6d. Longer training consistently improves the emergent zero-shot performance for both modalities across all datasets.

**Data augmentation for paired images.** During IMAGEBIND training, we augment images either using basic augmentation (cropping, color jitter) or strong augmentation that additionally applies RandAugment [Cubuk, 2020] and RandErase [Zhong, 2020]. We specify the augmentation parameters in section 6.4.5. Stronger augmentation helps depth classification when training on the small number of (image, depth) pairs from the SUN RGB-D dataset. However, for audio, strongly augmenting the video makes the task too challenging, leading to a significant drop of 34% on ESC.

**Depth specific design choices.** We vary the type of spatial crops used for training in table. 6.6f. Following CMC [Tian, 2019], we use two unaligned random crops from the corresponding image and depth pair vs. our default choice of using spatially aligned random crops. Contrary to CMC, we observe that random cropping severely degrades performance: more than 10% on
<table><caption>Table 6.7 – **Capacity of the audio and depth encoders** and their impact on performance. A stronger image encoder improves performance for both audio and depth tasks. As the number of (image, depth) pairs is small, a smaller encoder improves performance for depth. For audio classification, a larger encoder is better.</caption><thead><tr><th rowspan="2">Image Encoder</th><th colspan="2">Audio Encoder (ESC)</th><th colspan="2">Depth Encoder (SUN)</th></tr><tr><th>ViT-S</th><th>ViT-B</th><th>ViT-S</th><th>ViT-B</th></tr></thead><tbody><tr><td>ViT-B</td><td>52.8</td><td>56.7</td><td>30.7</td><td>26.7</td></tr><tr><td>ViT-H</td><td>54.8</td><td><b>60.3</b></td><td><b>33.3</b></td><td>29.5</td></tr></tbody></table>

<table><caption>Table 6.8 – **Different training procedures** for IMAGEBIND where we train *all encoders from scratch*. While joint training leads to worse performance, only updating the image encoder using the (image, text) loss, and jointly training the models is better.</caption><thead><tr><th></th><th>IN1K</th><th>ESC</th><th>SUN-D</th></tr></thead><tbody><tr><td>Joint training</td><td>14.1</td><td>7.1</td><td>12.4</td></tr><tr><td>Update image only w/ text</td><td>35.4</td><td>11.8</td><td>18.7</td></tr></tbody></table>

SUN-D. Unlike vanilla self-supervised learning, our image representations learned from image-text pairs are more semantic and thus spatially misaligned crops hurt performance. In table. 6.6g, we observe that RandomErase [Zhong, 2020] boosts performance on depth classification.

**Audio specific design choices.** We train for video-audio alignment using temporally aligned samples or unaligned samples and measure the final performance in table. 6.6h. Similar to the depth classification observation, temporally aligned samples lead to better performance. table. 6.6i shows that using frequency masking augmentation for audio also provides a small boost in performance.

**Capacity of the audio and depth encoders** and their impact of the classification performance is reported in table. 6.7. A smaller encoder for depth improves performance presumably because of the relatively small size of the (image, depth) dataset. Conversely, we observe that larger audio encoder improves the performance, particularly when paired with a high capacity image encoder.

**Training procedure.** We study the different choices in training procedure on images, text, depth, and audio. We jointly train all the encoders and do *not* initialize the image and text encoders. We use a subset of the image-text pairs from the LAION dataset [Schuhmann, 2021], and use the same datasets for depth and audio modalities as in section 6.5.2. Full training details are in section 6.4.5. We experiment with two settings in table. 6.8: (1) jointly training all encoders; and (2) stopping the gradient to the image encoder when aligning with audio and depth. Training the image representation solely from the text supervision (second setting) yields the best results not only for zero-shot image classification but also for the emergent zero-shot audio and depth classification.

**ImageBind to evaluate pretrained vision models** in table. 6.9. We initialize the vision encoder using a pretrained model and keep it fixed. We use image-paired data to align and train text, audio, and depth encoders. Compared to the supervised DeiT model, the self-supervised DINO model is better at emergent zero-shot classification on both depth and audio modalities. Moreover, the emergent zero-shot performance is not correlated with the pure vision performance
Chapter 6. Learning a shared embedding space for six modalities with ImageBind

<table><caption>Table 6.9 – ImageBind as an evaluation tool. We initialize (and fix) the image encoder with different methods and align other modalities. IMAGEBIND measures the impact of visual features on multimodal tasks. † trained with IN1K supervision.</caption><thead><tr><th></th><th>IN1K</th><th>VGGS</th><th>ESC</th><th>SUN-D</th><th>NYU-D</th></tr></thead><tbody><tr><td>DINO [Caron, 2021]</td><td>64.4</td><td>17.2</td><td>44.7</td><td>26.8</td><td>48.8</td></tr><tr><td>DeiT [Touvron, 2021b]</td><td>74.4<sup>†</sup></td><td>9.6</td><td>25.0</td><td>25.2</td><td>48.0</td></tr></tbody></table>

on ImageNet suggesting that these tasks measure different properties. IMAGEBIND can serve as a valuable tool to measure vision models’ strength on multimodal applications.

## 6.6 Discussion and Limitations

IMAGEBIND is a simple and practical way to train a joint embedding space using only image alignment. Our method leads to emergent alignment across all modalities which can be measured using cross-modal retrieval and text-based zero-shot tasks. We enable a rich set of compositional multimodal tasks across different modalities, show a way to evaluate pretrained vision models for non-vision tasks and ‘upgrade’ models like Detic and DALLE-2 to use audio. There are multiple ways to further improve IMAGEBIND. Our image alignment loss can be enriched by using other alignment data. Our embeddings are trained without a specific downstream task, and thus lag the performance of specialist models. More research into adapting general purpose embeddings for each task, including structured prediction tasks such as detection will be beneficial. Finally, new benchmarks, e.g. our emergent zero-shot task, to measure emergent abilities of multimodal models are essential to create new and exciting applications.
# Chapter 7

## Conclusion

Transformers are highly expressive and versatile, making them effective for various modalities. Initially, Transformers were primarily used for machine translation and NLP tasks, and it took a while for the computer vision research community to adopt them. However, the introduction of Vision Transformers by Dosovitskiy et al. [Dosovitskiy, 2021a] ignited a surge of research aimed at adapting these architectures to different vision tasks and addressing their limitations. In this manuscript, we delved into Vision Transformers and tackled a number of research questions relating to their design and application. First, we addressed their quadratic complexity limitation. Afterward, we explored the synergy between Masked Image Modeling and Transformers for self-supervised pre-training and image compression. Finally, we leveraged the universality of Transformers architecture to enable the learning of general-purpose representations for modalities with limited data. The methods we have proposed in this thesis have pushed the limit for state-of-the-art performance in a number of vision tasks and unlocked novel use cases by utilizing the power of Transformers. Below, we first re-discuss the challenges faced when first experimenting with Transformers, and how this has motivated some of our contributions. Then we will discuss some perspectives related to the topic of this thesis.

### 7.1 Summary of Contributions

#### 7.1.1 Reducing the quadratic complexity of Vision Transformers

The quadratic complexity of Transformers limits their applicability to vision tasks that require high image resolutions. We have experienced this first-hand with our effort to improve image retrieval with Transformers [El-Nouby, 2021b]. We have observed that off-the-shelf Vision Transformers can provide superior performance to their CNN counterparts for retrieval applications with low-resolution images. However, for particular object retrieval, where images are typically high in resolution, using Transformers to extract descriptors was extremely challenging. This motivated us to tackle this shortcoming by proposing XCiT [El-Nouby, 2021c]. In Chapter 3, we introduced cross-covariance attention, a novel transposed view of self-attention that has linear complexity in terms of image resolution but quadratic complexity with respect to the model's hidden dimension, which we can control. We then presented XCiT, a model architecture that relies on cross-covariance attention as its main building block. XCiT has a
simple columnar structure similar to a standard ViT model, with cross-covariance attention
used as a drop-in replacement for self-attention. XCiT has several advantages over ViT and
other efficient attention variants. It achieves the strongest reduction in memory consumption
and latency while retaining performance on par or stronger than its counterparts. At the time
it was published, XCiT is the first columnar attention-based architecture that achieved strong
performance for dense tasks such as object detection and semantic segmentation. With XCiT,
we can develop more efficient and accurate models for a wide range of vision tasks, addressing
the limitations of traditional self-attention-based architectures.

7.1.2 Masked Image Modeling as a powerful tool

Masked language modeling has become a popular paradigm for learning strong representa-
tions for NLP tasks following the success of BERT [Devlin, 2018]. This has motivated works like
BEiT [Bao, 2021] to adapt masked prediction to images. Therefore, Masked image modeling
initially arose as a tool for learning visual representations with efforts like BEiT [Bao, 2021],
MAE [He, 2021a], and SplitMask [El-Nouby, 2021a]. It has also been found as a useful tool for
other tasks like image generation [Chang, 2022] and image compression [El-Nouby, 2023].

**Self-supervised pre-training.** In Chapter 4 we investigated different self-supervised pre-training paradigms such as contrastive and joint embedding methods. We delved deeper into denoising autoencoding, in particular with masking as the form of noise, using Transformers. Specifically, we proposed a novel self-supervised denoising objective called SplitMask, which splits each image into disjoint sets and operates in a more sample-efficient manner compared to other competing methods like BEiT. We conducted an in-depth analysis of the different properties of SplitMask and popular joint embedding methods, such as DINO, to demonstrate the advantages of our proposed method. First, we showed that masked image modeling approaches, in general, and SplitMask, in particular, have much higher sample efficiency during pre-training compared to DINO and supervised approaches. Second, we observed that SplitMask can effectively learn using more diverse types of visual data compared to joint embedding methods, which in turn have a strong bias towards object-centric images. Our findings indicate that novel use cases can be enabled by methods such as SplitMask. For example, companies or individuals can train strong visual models relying only on their private collection of data, eliminating the dependence on any third-party large-scale datasets that are typically required for successful pre-training. Our results show the utility of masked image modeling as a sample efficient objective. However, the flip perspective of our observations indicate that this family of methods does not exhibit a strong scaling behavior with respect dataset size, which stand in contrast with language models like GPT [Brown, 2020] where immensely scaling the data was a key element of their success.

**Entropy Modeling.** In Chapter 5, we presented an interesting application of masked image modeling in the context of neural image compression. Initially, we revisited vector quantization, demonstrating that while it can deliver satisfactory outcomes for extremely low bit-rates, the
vocabulary sizes exhibit exponential growth as we increase the permitted bit-rates. This renders vector quantization impractical except within a limited range of low bit-rates. In response to this challenge, we proposed product quantization as a viable alternative. Product quantization partitions the latent into multiple sub-vectors, each of which is quantized independently using a moderately-sized vocabulary. This approach significantly simplifies the optimization problem and yields an improved scaling behavior in relation to bit-rate. Subsequently, we turned our attention to the role of the entropy model in influencing the compression rate. We introduced PQ-MIM, a Transformer pre-trained for masked patch prediction, with an objective closely mirroring those discussed in Chapter 4. Given a specific masked patch, PQ-MIM can serve as a robust entropy model, estimating the assignment probability of each symbol in the vocabulary. This estimation progressively improves with increasing the number of observed patches used for conditioning. Hence, we introduced the Quincunx pattern where an image is segmented into five subsets, following a generalized checkerboard pattern, effectively doubling the number of patches included in each subset. We demonstrated that our final model which combines product quantization and our MIM module with a Quincunx pattern, achieves robust rate-distortion performance. This paves the way for extreme compression use cases, enabling the compression of an image into an ASCII code equivalent to the size of a tweet.

7.1.3 Learning representations for data-scarce modalities

Finally, Chapter 6 introduced ImageBind, a novel approach for achieving strong zero-shot
and open-set recognition capabilities in multimodal models. Inspired by the success of con-
trastive vision and language models like CLIP, we investigate how to extend these abilities to
other modalities, such as audio, thermal, depth, and IMU. We leverage existing semantic align-
ment between the visual and textual modalities and propose a novel approach for aligning new
modalities to the visual modality. We utilize pre-trained vision-language models, such as CLIP,
which have been trained on large-scale collections of image-text pairs covering a wide range of
semantic concepts, to provide a supervisory signal to the other modalities. Since it is more fea-
sible to collect aligned visual data, we align the representations of the new modalities to that of
the vision-language model using a simple contrastive objective, such as InfoNCE. This approach
allows us to demonstrate that the resulting features for the new modalities are highly semantic,
leading to strong zero-shot performance. Additionally, we show that aligning all modalities in a
single shared embedding space enables interesting applications for manipulating representations
across modalities. For example, we showcase that descriptors from different modalities can be
aggregated via simple vector arithmetic, enabling advanced retrieval applications. Furthermore,
we demonstrate that ImageBind multimodal representations can power image generation using
audio input as well as object detection with audio queries. Overall, ImageBind provides a power-
ful framework for achieving strong zero-shot and open-set recognition capabilities in multimodal
models, opening up new avenues for research in this exciting field.
## 7.2 Open Problems and Future Directions

In this thesis, we have taken steps towards adapting and pushing the limits of Transformer architectures for various computer vision tasks as well as multimodality. While the community has done significant progress, many use cases and applications require further research to be unlocked. For example, we need more efficient attention for very long sequences as is the case for videos. Additionally, to further improve the visual representations, we require self-supervised approaches that are more scalable and general-purpose. Finally, taking multimodal learning to the next level by supporting fully multimodal generative models.

### 7.2.1 Efficient Attention for Video Data

Despite XCiT's compelling performance and reduced complexity for high-resolution images, as discussed in Chapter 3, the model's quadratic complexity in terms of hidden dimension poses challenges for scaling to larger models. Moreover, more efficient attention mechanisms is required for long-range video understanding and generation, where the sequence length can grow significantly compared to the high-resolution image scenario we focused on with XCiT.

**Advancing Representation Learning in Videos.** As evidenced by the success of joint embedding and masking-based methods such as SplitMask, self-supervised representation learning for images has seen significant advancements. The corresponding progress for video data has been slower. Videos intuitively contain a wealth of information about the world, including physics and inter-object relationships, yet unsupervised systems pre-trained on videos are yet to outperform those trained on static images. A possible reason is the limited context (16 or 32 frames) typically used to train video models due to computational limitations. To maximally exploit the informational richness of videos, we might need to consider longer contexts, which necessitates a significantly more efficient attention formulation and implementation.

**Enhancing Video Generation.** The field has witnessed a revolution in image generation [Ramesh, 2022a], which has been followed by promising models for video generation [Singer, 2022]. These video models, however, are not yet on par with their image counterparts. They heavily rely on interpolation techniques and separable spatial and temporal attention to circumvent the computational complexity of attending over long sequences of spatiotemporal patches. We believe for video generation, even more than representation learning, unlocking the expressive power of attention in exploiting long-range dependencies can be invaluable and it has the potential to significantly enhance the quality of generations.

### 7.2.2 Improving the scalability of Self-supervised Learning

In Chapter 4, we presented the efficacy of self-supervised methods such as SplitMask for the sample-efficient and robust pre-training of vision models. However, the denoising family of methods, including SplitMask, MAE [He, 2021b], and BEiT [Peng, 2022], often exhibit weak
linear probing performance, making finetuning generally necessary to reap the gains. This con-
trasts with joint embedding methods, which typically deliver strong off-the-shelf performance
without finetuning, but they also present challenges in terms of low sample efficiency, bias to-
wards object-centric images, and difficulty in scaling model parameters. A key research question
is to identify a family of models that can enjoy the benefits of both methods.

**Scaling in terms of Data.** Denoising autoencoding methods, particularly SplitMask and MAE, show promising scalability in terms of model parameters due to their strategy of dropping masked patches during pre-training. However, as discussed in Chapter 4 and confirmed by Singh et al. [Singh, 2023], these methods do not demonstrate a clear benefit from additional data, which is a significant limitation. An open research question is how to modify the training objectives of these models to improve their scalability with respect to data, while retaining their sample efficiency and other favorable properties.

**Enhancing Off-the-shelf Performance.** Another limitation of denoising autoencoder meth-
ods is their relatively weak off-the-shelf performance, likely due to the generative nature of the
training objective, which does not incentivize last-layer features to be highly discriminative, as
is the case for joint embedding methods. With the rise of general-purpose large language models
and interest in augmenting them with visual reasoning abilities, there is a growing need for un-
biased visual features that are not specifically tuned for a fixed set of categories, yet are highly
discriminative. Developing pre-training methods that overcome the drawbacks of both joint
embedding and denoising methods is a challenging and open research question that warrants
significant investigation.
# Chapter 8

## Résumé Substantiel

L'apprentissage profond a eu un impact considérable dans le domaine de la vision par ordinateur. Cet impact a révolutionné notre manière de percevoir, de comprendre et d'interagir avec le monde numérique. Les progrès rapides ont conduit au développement de nouveaux algorithmes et d'architectures qui ont continuellement fait avancer l'état de l'art dans les tâches de vision par ordinateur. Les avancées en matière d'apprentissage profond ont surpassé les méthodes classiques de vision par ordinateur et ont permis une large gamme d'applications telles que la conduite autonome, l'analyse de l'environnement et la gestion de l'information. Par exemple, la conduite autonome [Chen, 2015], l'imagerie et la vidéo [Ramesh, 2022a; Singer, 2022], ainsi que l'analyse d'images médicales [Shin, 2016]. L'un des principaux facteurs qui ont contribué au succès de l'apprentissage profond dans le domaine de la vision par ordinateur a été l'innovation dans les architectures de réseau [Shin, 2016; He, 2016; Szegedy, 2015; Hu, 2018; Wang, 2018]. Ces innovations ont abouti à des modèles qui peuvent mieux apprendre et représenter des modèles complexes à partir de données à grande échelle. De plus, elles ont permis le développement de modèles capables d'apprendre des représentations hiérarchiques, qui sont crucales pour comprendre la structure des données visuelles.

Les réseaux neuronaux convolutifs (CNN) [Fukushima, 1980; LeCun, 1989] sont depuis longtemps considérés comme l'architecture de référence pour les tâches de vision par ordinateur. Ils sont considérés comme l'architecture la mieux adaptée à ces tâches. Les CNN se caractérisent par leur biais inductif et leur efficacité d'échantillonnage, ce qui leur permet d'apprendre et de généraliser efficacement à partir d'un nombre limité d'exemples d'apprentissage. Leur capacité à apprendre les caractéristiques locales et à représenter de manière robuste les hiérarchies spatiales en fait l'architecture dominante pour les tâches de vision. Le succès des CNN peut être attribué à leur capacité à exploiter la structure spatiale inhérente aux images, ce qui leur permet d'apprendre efficacement des représentations significatives. Malgré leur efficacité, les mêmes biais inductifs et restrictions de conception qui rendent les CNN efficaces peuvent potentiellement limiter leur généralisation à mesure que la quantité de données d'apprentissage augmente. Par exemple, les CNN sont connus pour avoir un fort biais en faveur de la texture, ce qui peut limiter leur capacité à se généraliser à de nouveaux ensembles de données d'images diversifiées.

Le paysage de l'apprentissage profond a commencé à évoluer avec l'introduction de l'architecture Transformer [Vaswani, 2017b]. À l'origine conçue pour la traduction automatique et
les tâches de traitement du langage naturel (NLP), l'architecture Transformer s'est avérée très efficace pour capturer les dépendances à long terme et les modèles complexes dans les données séquentielles. L'innovation clé de cette architecture est le mécanisme d'auto-attention, qui permet au modèle d'évaluer dynamiquement l'importance des différents éléments d'entrée dans le contexte de la séquence entière. Le succès des Transformers dans les tâches de NLP a incité les chercheurs à explorer leur potentiel dans le domaine de la vision par ordinateur. Dans un premier temps, les mécanismes d'auto-attention ont été intégrés aux architectures CNN existantes, donnant ainsi naissance à des modèles hybrides [Wang, 2018] qui combinaient les avantages des deux approches. L'étape logique suivante de cette évolution a été le développement des transformateurs de vision (ViT) [Dosovitskiy, 2021a], qui s'éloignent considérablement des approches traditionnelles basées sur les CNN. Les modèles ViT sont entièrement composés de modules d'auto-attention et de perceptrons multicouches (MLP), éliminant complètement les couches convolutes. Malgré ce changement radical d'architecture, les modèles ViT ont obtenu des résultats de pointe dans une large gamme de tâches de vision par ordinateur, surpassant les performances de leurs homologues basés sur les CNN. De plus, les transformateurs de vision présentent des propriétés différentes de celles des CNN, notamment une plus grande robustesse face aux occlusions et aux perturbations, ainsi qu'un biais moins prononcé en faveur de la texture [Naseer, 2021]. Ces propriétés font des ViT des candidats prometteurs pour améliorer les performances de généralisation dans de nombreuses tâches visuelles.

Les transformateurs ont ouvert de vastes possibilités de recherche dans le domaine de la vision par ordinateur. L'expressivité et la généralité des modèles Transformer les rendent très adaptables à de nombreuses tâches et domaines. De plus, leur capacité à modéliser des dépendances à longue portée leur permet de capturer de manière plus efficace le contexte global et les relations complexes au sein des données visuelles, par rapport aux CNN. Dans cette thèse, nous plongeons plus profondément dans le monde des Transformers, explorant leur potentiel et leurs applications dans diverses tâches de vision par ordinateur. Nous examinerons les facteurs qui ont contribué à leur succès et chercherons des moyens d'améliorer leur adaptation aux tâches de vision par ordinateur. De plus, nous explorerons comment l'unification des architectures à travers les modalités peut conduire à des approches novatrices pour fournir une supervision aux modalités disposant de peu de ressources, en s'appuyant sur des modèles de vision solides.

## 8.1 Défis

Les transformateurs ouvrent effectivement de nouvelles possibilités d'apprentissage pour obtenir des représentations visuelles plus puissantes grâce à l'apprentissage supervisé et non supervisé. Ils permettent également de développer des approches plus homogènes pour l'apprentissage multimodal, c'est-à-dire l'apprentissage à partir de différentes modalités (par exemple, images et textes) de manière cohérente. Dans ce manuscrit, nous nous attaquons à plusieurs défis liés à ces sujets passionnants.
### 8.1.1 Complexité computationnelle

L'un des principaux défis posés par les transformateurs, tels qu'ils ont été introduits par VASWANI et al. [Vaswani, 2017b], est la complexité quadratique en fonction de la longueur de la séquence. Le mécanisme d'auto-attention utilisé par les transformateurs nécessite le calcul des interactions entre chaque paire d'éléments de la séquence d'entrée, ce qui se traduit par une complexité de l'ordre de $O(N^2)$, où N est la longueur de la séquence. Dans le domaine de la vision par ordinateur, cette complexité pose un défi pour l'application directe des transformateurs aux images haute résolution, qui sont souvent nécessaires pour des tâches telles que la détection d'objets, la segmentation sémantique et la compression d'images. Pour surmonter cette limitation, il est nécessaire d'innover au niveau architectural afin de réduire le coût de calcul des transformateurs pour les tâches liées aux images haute résolution.

### 8.1.2 Apprentissage de la représentation à grande échelle et efficace en termes d'échantillons

Les transformateurs excellent dans la capture des dépendances à longue portée et peuvent apprendre efficacement les représentations hiérarchiques. À l'instar de leur impact sur l'apprentissage des représentations avec des approches telles que BERT [Devlin, 2018] et GPT [Radford, 2018], les transformateurs ont le potentiel de révolutionner le pré-entraînement auto-supervisé pour les tâches de vision. Cependant, pour réaliser pleinement ce potentiel, il est essentiel de développer des méthodes plus efficaces en termes d'échantillonnage qui peuvent être mises à l'échelle. Des méthodes plus efficaces en termes d'échantillons et qui peuvent s'adapter plus facilement en termes de données et de calcul par rapport aux méthodes conjointes existantes. Le pouvoir d'expression de l'architecture Transformer, lorsqu'elle est utilisée en conjonction avec des méthodes plus génériques et moins biaisées, est un atout majeur. Avec des objectifs de débruitage plus génériques et moins biaisés, elle a le potentiel de fournir l'efficacité et l'évolutivité nécessaires pour améliorer considérablement le pré-entraînement auto-supervisé des modèles de vision.

### 8.1.3 Apprentissage multimodal

En tant qu'architecture universelle, le modèle Transformer a été appliqué avec succès à diverses modalités. Cette polyvalence ouvre la voie à des principes de conception et à des composants communs qui peuvent faciliter l'apprentissage multimodal. En utilisant les transformateurs comme base pour l'apprentissage multimodal, nous pouvons développer des méthodes qui intègrent efficacement les informations provenant de diverses modalités et faciliter le transfert transparent de connaissances entre différentes tâches et domaines. Cependant, alors qu'il peut y avoir une abondance de données pour certaines modalités, d'autres peuvent souffrir d'une grave pénurie de données. Dans de tels cas, le transfert de connaissances rendu possible par l'utilisation d'un cadre commun tel que Transformers peut avoir un impact significatif sur la qualité des données. En tirant parti des différentes modalités et en transférant les connaissances acquises d'un domaine à l'autre, nous pouvons améliorer la performance des modèles dans des domaines
où la rareté des données est un problème majeur.

## 8.2 Présentation et contributions

Le manuscrit commence par une discussion sur le contexte et la littérature pertinente dans le chapitre 2. Ensuite, nous approfondissons les détails de chacune de nos quatre principales contributions dans les chapitres suivants, comme indiqué ci-dessous.

### 8.2.1 Maîtriser la complexité quadratique des transformateurs de vision

Dans le chapitre 3, nous étudions la complexité informatique des transformateurs de vision [Dosovitskiy, 2021a]. Plus précisément, nous constatons que l'opération d'auto-attention, qui est au cœur de l'architecture des transformateurs Transformer [Vaswani, 2017b], présente une croissance quadratique en fonction de la résolution de l'image. Cette complexité peut rendre les transformateurs de vision moins efficaces, en particulier lorsqu'il s'agit de traiter des images à haute résolution, comme c'est souvent le cas dans les applications de vision par ordinateur, telles que la segmentation sémantique et la détection d'objets. Pour relever ce défi, nous proposons une nouvelle formulation alternative de l'opération d'auto-attention, que nous appelons *Cross-Covariance Attention* (XCA) [El-Nouby, 2021c]. L'attention à covariance croisée présente une complexité linéaire par rapport à la taille de l'entrée et peut être intégrée de manière transparente en remplacement de l'auto-attention dans les transformateurs de vision. Nous présentons XCIiT, une nouvelle architecture pour la vision par ordinateur qui utilise XCA comme élément central. Nous démontrons que XCIiT offre des améliorations significatives en termes de mémoire et de débit tout en conservant les excellentes performances des transformateurs de vision.

### 8.2.2 Pré-entraînement auto-supervisé efficace avec Transformer

L'architecture Transformer a été un catalyseur pour l'innovation en matière de pré-entraînement auto-supervisé dans le domaine du traitement du langage naturel. Elle a conduit au développement de modèles influents tels que BERT [Devlin, 2018] et GPT [Radford, 2018]. Dans le domaine de la vision par ordinateur, des méthodes telles que [Chen, 2020c; He, 2020; Caron, 2021] se sont imposées grâce à leur grande adaptabilité et à leurs performances exceptionnelles. Elles ont démontré leur compétitivité par rapport aux méthodes supervisées pour le fine-tuning des images. Cependant, les méthodes d'intégration conjointe présentent certaines limitations, notamment leur dépendance à l'égard de techniques d'augmentation de données conçues manuellement et adaptées à ImageNet [Deng, 2009b]. De plus, ces méthodes peuvent souffrir de problèmes de taille de modèle, comme l'a souligné [Chen, 2021b]. Motivés par le succès de BERT dans le domaine du NLP, dans le chapitre 4, nous étudions l'efficacité des méthodes de débruitage et d'auto-codage lorsqu'elles sont utilisées en conjonction avec des transformateurs de vision pour le pré-entraînement auto-supervisé. Tout d'abord, nous proposons SplitMask [El-Nouby, 2021a], une nouvelle méthode de pré-entraînement auto-supervisée basée sur la modélisation d'images masquées. De plus, nous constatons que SplitMask offre une meilleure efficacité d'échantillonnage et peut être utilisée pour apprendre des représentations robustes en utilisant des ensembles
de données de plusieurs ordres de grandeur plus petits que ceux requis par les méthodes d'intégration conjointe. En outre, nous observons que SplitMask est bien adapté à l'apprentissage avec une plus large gamme de données visuelles car il n'est pas biaisé par une distribution spécifique d'images, contrairement aux méthodes d'intégration conjointe qui sont généralement centrées sur l'objet d'ImageNet.

### 8.2.3 Modélisation d'images masquées pour une meilleure compression d'images

La compression neuronale des images s'est révélée être une alternative prometteuse aux codecs traditionnels en raison de sa qualité d'image supérieure sur le plan perceptif et psychovisuel. En général, les systèmes de compression d'images neuronales se composent de trois éléments principaux : un encodeur et un décodeur neuronaux profonds, qui apprennent à mapper l'image et sa représentation latente inverse ; un quantificateur, qui fait correspondre les représentations latentes continues à un ensemble de symboles discrets ; et un modèle d'entropie, qui exploite les redondances dans l'ensemble de symboles discrets pour réduire la longueur finale du flux binaire. Dans le chapitre 5, nous proposons plusieurs contributions. Tout d'abord, nous adoptons XCI-T pour concevoir le codeur et le décodeur. Ensuite, nous utilisons la quantification par produit au lieu de la quantification vectorielle. Enfin, nous proposons un nouveau modèle d'entropie pour la compression d'images neuronales basé sur la modélisation d'images masquées. Notre méthode, PQ-MIM [El-Nouby, 2023], permet d'obtenir une forte réduction des débits par rapport aux méthodes basées sur la fréquence, tout en offrant des avantages supplémentaires par rapport aux méthodes autorégressives qui peuvent être prohibitives pour les images à haute résolution. Ainsi, PQ-MIM permet une compression extrême des images à la taille d'un tweet ou d'un SMS.

### 8.2.4 ImageBind pour l'apprentissage d'un espace d'intégration partagé pour six modalités

Les transformateurs ont émergé comme une architecture polyvalente offrant des performances exceptionnelles dans différentes modalités, telles que le texte [Devlin, 2018; Radford, 2018], les images [Dosovitskiy, 2021a; Touvron, 2020], la vidéo [Gedas Bertasius, 2021; Tong, 2022], l'audio [Xu, 2022], et les graphes [Yun, 2019], entre autres. Cela ouvre des perspectives passionnantes pour le développement de systèmes multimodaux avec des composants partagés et des conceptions similaires. Cependant, l'approche prédominante pour former ces modalités rePOSE SUR L'apprentissage supervisé, qui consiste à apprendre à partir d'entrées sensorielles un ENSEMBLE D'étiquettes catégoriques. Malheureusement, l'apprentissage supervisé est limité par LA DIFFICULTÉ DE COLLECTER efficacement ET DE MANIÈRE ÉVOLUTIVE des ANNOTATIONS. POUR SURMONTER CE DÉFI, L'apprentissage FAIBLEMENT supervisé est apparu comme un paradigme alternatif qui se base sur la collecte de grandes quantités de données avec des étiquettes bruitées, qui peuvent ÊTRE PLUS FACILES À ACQUÉRIR. CE PARADIGME a connu un immense succès dans l'apprentissage de représentations visuelles, alimenté par des collections massives de paires d'images et de textes PROVENANT DU WEB OUVERT [Joulin, 2016; Radford, 2021; Zhai, 2022]. CEPENDANT, LA GÉNÉRATION DE COLLECTIONS SIMILAIRES POUR D'autres modalités telles que L'AUDIO, LES IMAGES THERMiques ET LES
images de profondeur est beaucoup plus difficile. Dans le chapitre 6, nous abordons ce problème en introduisant ImageBind, une nouvelle méthode d'entraînement qui permet d'encoder six modalités différentes dans un espace latent partagé, en tirant parti des performances élevées des modèles de vision et de langage existants.

Le manuscrit se conclut par le chapitre 7, qui résume nos principales conclusions et idées, tout en abordant les limitations de nos approches et en proposant des pistes pour de futures recherches.
# Appendix

## A Additional results for XCiT

### A.1 More XCiT models

We present additional results for our XCiT models in Table A.1. We include performance of 384×384 images using a 16×16 patch size as well as results for images with 224×224 resolution using patch size of 8×8.

<table><caption>TABLE A.1 – ImageNet-1k top-1 accuracy of XCiT for additional combinations of image and patch sizes.</caption><thead><tr><th rowspan="2">Models</th><th rowspan="2">Depth</th><th rowspan="2">d</th><th rowspan="2">#Blocks</th><th rowspan="2">#params</th><th colspan="4">P = (16 × 16)</th><th colspan="4">P = (8 × 8)</th></tr><tr><th>GFLOPs</th><th>@224</th><th>@224Υ</th><th>@384↑</th><th>GFLOPs</th><th>@224</th><th>@224Υ</th><th>@384↑</th></tr></thead><tbody><tr><td>XCiT-N12</td><td>12</td><td>128</td><td>4</td><td>3M</td><td>0.5</td><td>69.9</td><td>72.2</td><td>75.4</td><td>2.1</td><td>73.8</td><td>76.3</td><td>77.8</td></tr><tr><td>XCiT-T12</td><td>12</td><td>192</td><td>4</td><td>7M</td><td>1.2</td><td>77.1</td><td>78.6</td><td>80.9</td><td>4.8</td><td>79.7</td><td>81.2</td><td>82.4</td></tr><tr><td>XCiT-T24</td><td>24</td><td>192</td><td>4</td><td>12M</td><td>2.3</td><td>79.4</td><td>80.4</td><td>82.6</td><td>9.2</td><td>81.9</td><td>82.6</td><td>83.7</td></tr><tr><td>XCiT-S12</td><td>12</td><td>384</td><td>8</td><td>26M</td><td>4.8</td><td>82.0</td><td>83.3</td><td>84.7</td><td>18.9</td><td>83.4</td><td>84.2</td><td>85.1</td></tr><tr><td>XCiT-S-24</td><td>24</td><td>384</td><td>8</td><td>48M</td><td>9.1</td><td>82.6</td><td>83.9</td><td>85.1</td><td>36.0</td><td>83.9</td><td>84.9</td><td>85.6</td></tr><tr><td>XCiT-M24</td><td>24</td><td>512</td><td>8</td><td>84M</td><td>16.2</td><td>82.7</td><td>84.3</td><td>85.4</td><td>63.9</td><td>83.7</td><td>85.1</td><td>85.8</td></tr><tr><td>XCiT-L24</td><td>24</td><td>768</td><td>16</td><td>189M</td><td>36.1</td><td>82.9</td><td>84.9</td><td>85.8</td><td>142.2</td><td>84.4</td><td>85.4</td><td>86.0</td></tr></tbody></table>

### A.2 Image Retrieval

**Context of this study.** Vision-based retrieval tasks such as landmark or particular object retrieval have been dominated in the last years by methods extracting features from high-resolution images. Traditionally, the image description was obtained as the aggregation of local descriptors, like in VLAD [Jégou, 2012]. Most of the modern methods now rely on convolutional neural networks [Berman, 2019; Gordo, 2017; Tolias, 2016b]. In a recent paper, El-Nouby et al. [El-Nouby, 2021b] show promising results with vision transformers, however they also underline the inherent scalability limitation associated with the fact that ViT models do not scale well with image resolution. Therefore, it cannot compete with convolutional neural networks whose performance readily improve with higher resolution images. Our XCiT models do not suffer from this limitation: our models scale linearly with the number of pixels, like convnets, and therefore makes it possible to use off-the-shelf methods initially developed for retrieval with high-resolution images.
**Datasets and evaluation measure** In each benchmark, a set of query images is searched in a database of images and the performance is measured as the mean average precision.

The Holidays [Jégou, 2008] dataset contains images of 500 different objects or scenes. We use the version of the dataset where the orientation of images (portrait or landscape) has been corrected. Oxford [Philbin, 2007] is a dataset of building images, which corresponds to famous landmark in Oxford. A similar dataset has been produced for famous monuments in Paris and referred to as Paris6k [Chum, 2007].

<table><caption>TABLE A.2 – The basic statistics on the image retrieval datasets.</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="2">number of images</th><th rowspan="2">nb of instances</th></tr><tr><th>database</th><th>queries</th></tr></thead><tbody><tr><td>Holidays</td><td>1491</td><td>500</td><td>500</td></tr><tr><td>R-Oxford</td><td>4993</td><td>70</td><td>26</td></tr></tbody></table>

We use the revisited version of the Oxford benchmark [Radenović, 2018a], which breaks down the evaluation into easy, medium and hard categories. We report results on the "medium" and "hard" settings, as we observed that the ordering of techniques does not change under the easy measures.

**Image representation : global and local description with XCIT** We consider three existing methods to extract an image vector representations from the pre-trained XCIT models. Note that to the best of our knowledge, for the first time we extract local features from the output layer of a transformer layer, and treat them as patches fed to traditional state-of-the-art methods based on matching local descriptors or CNN.

**CLS token.** Similar to EL-NOUBY et al. [El-Nouby, 2021b] with ViT, we use the final vector as the image descriptor. In this context, the introduction of class-attention layers can be regarded as a way to learn the aggregation method.

**VLAD.** We treat the patches before the class-attention layers as individual local descriptors, and aggregate them into a higher-dimensional vector by employing the Vector of locally aggregated Descriptors [Jégou, 2012].

**AMSK.** We also apply the aggregated selective match kernel from Tolias et al. [Tolias, 2016a]. This method was originally introduced for local descriptors, but got adapted to convolutional networks. To the best of our knowledge this is the state of the art on several benchmarks [Tolias, 2020].

For all these methods, we use the models presented in our main paper, starting from the version fine-tuned at resolution 384×384. By default the resolution is 768. This is comparable to the choice adopted in the literature for ResNet (e.g., 800 in the work by Berman et al. [Berman, 2019]).
**Experimental setting : Image retrieval with models pretrained on Imagenet1k only**
We only consider models pre-trained on Imagenet-1k. Note that the literature reports signifi-
cant improvement when learning or fine-tuning networks [Radenović, 2018b; Tolias, 2020] on
specialized datasets (e.g., of buildings for Oxford5k and Paris6k). We consider only XCiT-S12
models, since they have a number of parameters comparable to that of ResNet-50. We report
the results in Table A.3.

**Scaling resolution.** As expected increasing the resolution with XCiT improves the perfor-
mance steadily up to resolution 768. This shows that our models are very tolerant to resolution
changes considering that they have been fine-tuned at resolution 384. The performance starts
to saturates at resolution 1024, which led us to keep 784 as the pivot resolution.

**Self-supervision.** The networks XCiT pre-trained with self-supervision achieve a comparati-
vely better performance than their supervised counterpart on Holidays, however, we have the
opposite observation for *R*Oxford.

**Impact of Image description.** We adopt the class-token as the descriptor, and in our experiments we verified that this aggregation method is better than average and GeM pooling [Boureau, 2010; Radenović, 2018b]. In Table A.3 one can see there is a large benefit in employing a patch based method along with our XCiT transformers : XCiT-VLAD performs significantly better than the CLS token, likely thanks to the higher dimensionality. This is further magnified with AMSK, where we obtain results approaching the absolute state of the art on Holidays, despite a sub-optimal training setting for image retrieval. This is interesting since our method has not been fine-tuned for retrieval tasks and we have not been adapted in any significant way beyond applying off-the-shelf this aggregation technique. A direct comparison with ResNet-50 shows that our XCiT method obtains competitive results in this comparable setting, slightly below the ResNet-50 on *R*Oxford but significantly better on Holidays.
<table><thead><tr><th rowspan="2">Base model</th><th rowspan="2">parameters</th><th colspan="2">ROxford5k (mAP)</th><th rowspan="2">Holidays (mAP)</th></tr><tr><th>Medium</th><th>Hard</th></tr></thead><tbody><tr><td colspan="5" style="text-align: center; font-weight: bold;">XCiT- class token</td></tr><tr><td>XCiT-S12/16</td><td></td><td>30.1</td><td>8.7</td><td>86.0</td></tr><tr><td>XCiT-S12/8</td><td></td><td>33.2</td><td>12.1</td><td>86.4</td></tr><tr><td>XCiT-S12/16</td><td>resolution 224</td><td>12.7</td><td>2.4</td><td>71.5</td></tr><tr><td>XCiT-S12/16</td><td>resolution 384</td><td>20.1</td><td>4.6</td><td>83.4</td></tr><tr><td>XCiT-S12/16</td><td>resolution 512</td><td>26.6</td><td>5.8</td><td>84.6</td></tr><tr><td>XCiT-S12/16</td><td>resolution 768</td><td>30.1</td><td>8.7</td><td>86.0</td></tr><tr><td>XCiT-S12/16</td><td>resolution 1024</td><td>30.3</td><td>11.2</td><td>86.3</td></tr><tr><td>XCiT-S12/16</td><td>self-supervised DINO</td><td>35.1</td><td>11.9</td><td>87.3</td></tr><tr><td>XCiT-S12/8</td><td>self-supervised DINO</td><td>30.9</td><td>7.9</td><td>88.3</td></tr><tr><td colspan="5" style="text-align: center; font-weight: bold;">XCiT- VLAD</td></tr><tr><td>XCiT-S12/16</td><td>k=256</td><td>36.6</td><td>11.6</td><td>89.9</td></tr><tr><td>XCiT-S12/16</td><td>k=1024</td><td>40.0</td><td>13.0</td><td>90.7</td></tr><tr><td colspan="5" style="text-align: center; font-weight: bold;">XCiT- ASMK</td></tr><tr><td>XCiT-S12/8</td><td>k=1024</td><td>36.5</td><td>9.4</td><td>90.4</td></tr><tr><td>XCiT-S12/8</td><td>k=65536</td><td>42.0</td><td>12.9</td><td>92.3</td></tr><tr><td>XCiT-S12/16</td><td>k=1024</td><td>35.2</td><td>11.5</td><td>90.4</td></tr><tr><td>XCiT-S12/16</td><td>k=65536</td><td>40.0</td><td>15.0</td><td>92.0</td></tr><tr><td colspan="5" style="text-align: center; font-weight: bold;">ResNet-50 - ASMK</td></tr><tr><td>Resnet50</td><td>k=1024</td><td>41.6</td><td>14.6</td><td>86.0</td></tr><tr><td>Resnet50</td><td>k=65536</td><td>41.9</td><td>14.5</td><td>87.9</td></tr><tr><td>Multigrain-resnet50</td><td>k=1024</td><td>32.9</td><td>9.4</td><td>87.9</td></tr></tbody></table>

TABLE A.3 – **Instance retrieval experiments.** The default resolution is 768. The default class token size is 128 dimensions. The "local descriptor" representation extracted from the activations is in 128 dimensions. To our knowledge the state of the art with ResNet-50 on Holidays with Imagenet pre-training only is the Multigrain method [Berman, 2019], which achieves mAP=92.5%. Here we compare against this method under the same training setting, i.e., off-the-shelf network pre-trained on Imagenet1k only and with the same training procedure and resolution. We refer the reader to Tolias et al. [Tolias, 2020] for the state of the art on ROxford, which involves some training on the target domain with images depicted building and fine-tuning at the target resolution.
References

[Agustsson, 2017] E. AGUSTSSON, F. MENTZER, M. TSCHANNEN, L. CAVIGELLI, R. TIMOFTE, L. BENINI et L. v. GOOL, « Soft-to-hard vector quantization for end-to-end learning compressible representations », *Advances in Neural Information Processing Systems*, 2017 (cf. p. 54, 56).

[Agustsson, 2019] E. AGUSTSSON, M. TSCHANNEN, F. MENTZER, R. TIMOFTE et L. van GOOL, « Generative Adversarial Networks for Extreme Learned Image Compression », *International Conference on Computer Vision*, 2019 (cf. p. 54, 56).

[Ainslie, 2020] J. AINSLIE, S. ONTANON, C. ALBERTI, V. CVICEK, Z. FISHER, P. PHAM, A. RAVULA, S. SANGHAI, Q. WANG et L. YANG, « ETC : Encoding Long and Structured Inputs in Transformers », *Conference on Empirical Methods in Natural Language Processing*, 2020 (cf. p. 19).

[Alayrac, 2022] J.-B. ALAYRAC, J. DONAHUE, P. LUC, A. MIECH, I. BARR, Y. HASSON, K. LENC, A. MENSCH, K. MILLICAN, M. REYNOLDS et al., « Flamingo : a visual language model for few-shot learning », *arXiv preprint arXiv:2204.14198*, 2022 (cf. p. 15, 71, 72).

[Alayrac, 2020] J.-B. ALAYRAC, A. RECASENS, R. SCHNEIDER, R. ARANDJELOVIC, J. RAMAPURAM, J. DE FAUW, L. SMAIRA, S. DIELEMAN et A. ZISSERMAN, « Self-Supervised MultiModal Versatile Networks. », *Advances in Neural Information Processing Systems*, 2020 (cf. p. 72).

[Alemi, 2018] A. ALEMI, B. POOLE, I. FISCHER, J. DILLON, R. A. SAUROUS et K. MURPHY, « Fixing a broken ELBO », *International Conference on Learning Representations*, 2018 (cf. p. 56).

[Antonini, 1992] M. ANTONINI, M. BARLAUD, P. MATHIEU et I. DAUBECHIES, « Image coding using wavelet transform », t. 1, n° 2, p. 205-220, 1992 (cf. p. 54).

[Arandjelovic, 2017] R. ARANDJELOVIC et A. ZISSERMAN, « Look, listen and learn », *International Conference on Computer Vision*, 2017 (cf. p. 71, 72).

[Arnab, 2021] A. ARNAB, M. DEHGHANI, G. HEIGOLD, C. SUN, M. LUČIĆ et C. SCHMID, « Vivit : A video vision transformer », *arXiv preprint arXiv:2103.15691*, 2021 (cf. p. 17).

[Asano, 2019a] Y. M. ASANO, C. RUPPRECHT et A. VEDALDI, « A critical analysis of self-supervision, or what we can learn from a single image », *arXiv preprint arXiv:1904.13132*, 2019 (cf. p. 40).

[Asano, 2019b] Y. M. ASANO, C. RUPPRECHT et A. VEDALDI, « Self-labelling via simultaneous clustering and representation learning », *arXiv preprint arXiv:1911.05371*, 2019 (cf. p. 14, 40).

[Assran, 2021] M. ASSRAN, M. CARON, I. MISRA, P. BOJANOWSKI, A. JOULIN, N. BALLAS et M. RABBAT, « Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples », *arXiv preprint arXiv:2104.13963*, 2021 (cf. p. 41).

[Asuni, 2014] N. ASUNI et A. GIACHETTI, « TESTIMAGES : A large-scale archive for testing visual devices and basic image processing algorithms (SAMPLING 1200 RGB set) », *STAG : Smart Tools and Apps for Graphics*, 2014 (cf. p. 64).

[Atito, 2021] S. ATITO, M. AWAIS et J. KITTLER, « Sit : Self-supervised vision transformer », *arXiv preprint arXiv:2104.03602*, 2021 (cf. p. 39).

[Ba, 2016] J. L. BA, J. R. KIROS et G. E. HINTON, « Layer normalization », *arXiv preprint arXiv:1607.06450*, 2016 (cf. p. 23, 24).

[Bachmann, 2022] R. BACHMANN, D. MIZRAHI, A. ATANOV et A. ZAMIR, « MultiMAE : Multi-modal Multi-task Masked Autoencoders », *arXiv preprint arXiv:2204.01678*, 2022 (cf. p. 71, 79).

[Bahdanau, 2014] D. BAHDANAU, K. CHO et Y. BENGIO, « Neural machine translation by jointly learning to align and translate », *arXiv preprint arXiv:1409.0473*, 2014 (cf. p. 8).
[Bain, 2021] M. BAIN, A. NAGRANI, G. VAROL et A. ZISSERMAN, « Frozen in time : A joint video and image encoder for end-to-end retrieval », *arXiv preprint arXiv:2104.00650*, 2021 (cf. p. 78).

[Ballé, 2017] J. BALLÉ, V. LAPARRA et E. P. SIMONCELLI, « End-to-end Optimized Image Compression », *International Conference on Learning Representations*, 2017 (cf. p. 56).

[Ballé, 2018] J. BALLÉ, D. MINNEN, S. SINGH, S. J. HWANG et N. JOHNSTON, « Variational image compression with a scale hyperprior », *International Conference on Learning Representations*, 2018 (cf. p. 56, 61, 64).

[Bao, 2021] H. BAO, L. DONG et F. WEI, « BEiT : BERT Pre-Training of Image Transformers », *arXiv preprint arXiv:2106.08254*, 2021 (cf. p. 38, 39, 41, 42, 46, 48, 50, 51, 90).

[Bao, 2022] H. BAO, L. DONG et F. WEI, « BEiT : BERT Pre-Training of Image Transformers », *International Conference on Learning Representations*, 2022 (cf. p. 54).

[Bardes, 2021] A. BARDES, J. PONCE et Y. LECUN, « Vicreg : Variance-invariance-covariance regularization for self-supervised learning », *arXiv preprint arXiv:2105.04906*, 2021 (cf. p. 14, 40).

[Bellard, ] F. BELLARD, *BPG Image format* (cf. p. 61, 64).

[Bello, 2021] I. BELLO, « LambdaNetworks : Modeling long-range interactions without attention », *arXiv preprint arXiv:2102.08602*, 2021 (cf. p. 11, 20, 33).

[Beltagy, 2020] I. BELTAGY, M. E. PETERS et A. COHAN, « Longformer : The long-document transformer », *arXiv preprint arXiv:2004.05150*, 2020 (cf. p. 9, 19).

[Bengio, 2007] Y. BENGIO, P. LAMBLIN, D. POPOVICI et H. LAROCHELLE, « Greedy layer-wise training of deep networks », *Advances in neural information processing systems*, 2007, p. 153-160 (cf. p. 39).

[Bengio, 2013] Y. BENGIO, N. LÉONARD et A. COURVILLE, « Estimating or propagating gradients through stochastic neurons for conditional computation », t. arXiv:1308.3432, 2013 (cf. p. 62).

[Berman, 2019] M. BERMAN, H. JÉGOU, A. VEDALDI, I. KOKKINOS et M. DOUZE, « MultiGrain : a unified image embedding for classes and instances », *arXiv preprint arXiv:1902.05509*, 2019 (cf. p. 100, 101, 103).

[Bertasius, 2021] G. BERTASIUS, H. WANG et L. TORRESANI, « Is Space-Time Attention All You Need for Video Understanding? », *arXiv preprint arXiv:2102.05095*, 2021 (cf. p. 17).

[Bińkowski, 2018] M. BIŃKOWSKI, D. J. SUTHERLAND, M. ARBEL et A. GRETTON, « Demystifying MMD GANs », *International Conference on Learning Representations*, 2018 (cf. p. 64).

[Bossard, 2014] L. BOSSARD, M. GUILLAUMIN et L. VAN GOOL, « Food-101 – Mining Discriminative Components with Random Forests », *European Conference on Computer Vision*, 2014 (cf. p. 45, 46).

[Bottou, 1998] L. BOTTOU, P. HAFFNER, P. G. HOWARD, P. SIMARD, Y. BENGIO et Y. LE CUN, « High quality document image compression with “DjVu” », *Journal of Electronic Imaging*, t. 7, n° 3, p. 410-425, 1998 (cf. p. 56).

[Boureau, 2010] Y.-L. BOUREAU, J. PONCE et Y. LECUN, « A Theoretical Analysis of Feature Pooling in Visual Recognition », *International Conference on Machine Learning*, 2010 (cf. p. 102).

[Brabandere, 2016] B. D. BRABANDERE, X. JIA, T. TUYTELAARS et L. V. GOOL, « Dynamic Filter Networks », *Advances in Neural Information Processing Systems*, 2016 (cf. p. 20).

[Brock, 2021] A. BROCK, S. DE, S. L. SMITH et K. SIMONYAN, « High-Performance Large-Scale Image Recognition Without Normalization », *arXiv preprint arXiv:2102.06171*, 2021 (cf. p. 8, 18, 28, 29).

[Brown, 2020] T. B. BROWN, B. MANN, N. RYDER, M. SUBBIAH, J. KAPLAN, P. DHARIWAL, A. NEELAKANTAN, P. SHYAM, G. SASTRY, A. ASKELL et al., « Language models are few-shot learners », *preprint arXiv:2005.14165*, 2020 (cf. p. 90).

[Bruna, 2016] J. BRUNA, P. SPRECHMANN et Y. LECUN, « Super-resolution with deep convolutional sufficient statistics », *International Conference on Learning Representations*, 2016 (cf. p. 54).

[Carion, 2020] N. CARION, F. MASSA, G. SYNNAEVE, N. USUNIER, A. KIRILLOV et S. ZAGORUYKO, « End-to-End Object Detection with Transformers », *European Conference on Computer Vision*, 2020 (cf. p. 11, 32, 35, 37).

[Caron, 2018] M. CARON, P. BOJANOWSKI, A. JOULIN et M. DOUZE, « Deep clustering for unsupervised learning of visual features », *Proceedings of the European Conference on Computer Vision (ECCV)*, 2018, p. 132-149 (cf. p. 14, 40).

[Caron, 2019] M. CARON, P. BOJANOWSKI, J. MAIRAL et A. JOULIN, « Unsupervised pre-training of image features on non-curated data », *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2019, p. 2959-2968 (cf. p. 40).
[Caron, 2020a] M. CARON, I. MISRA, J. MAIRAL, P. GOYAL, P. BOJANOWSKI et A. JOULIN, « Unsupervised learning of visual features by contrasting cluster assignments », *arXiv preprint arXiv:2006.09882*, 2020 (cf. p. 14, 37).

[Caron, 2020b] M. CARON, I. MISRA, J. MAIRAL, P. GOYAL, P. BOJANOWSKI et A. JOULIN, « Unsupervised learning of visual features by contrasting cluster assignments », *arXiv preprint arXiv:2006.09882*, 2020 (cf. p. 40).

[Caron, 2021] M. CARON, H. TOUVRON, I. MISRA, H. JÉGOU, J. MAIRAL, P. BOJANOWSKI et A. JOULIN, « Emerging properties in self-supervised vision transformers », *arXiv preprint arXiv:2104.14294*, 2021 (cf. p. 4, 14, 18, 25, 31, 37, 40, 41, 46, 48, 49, 88, 97).

[Carreira, 2017] J. CARREIRA et A. ZISSERMAN, « Quo vadis, action recognition ? a new model and the kinetics dataset », *proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2017, p. 6299-6308 (cf. p. 37, 75).

[Chang, 2022] H. CHANG, H. ZHANG, L. JIANG, C. LIU et W. T. FREEMAN, « MaskGIT : Masked Generative Image Transformer », *Computer Vision and Pattern Recognition*, 2022 (cf. p. 54, 57, 58, 60, 65, 66, 90).

[Chappelier, 2006] V. CHAPPELIER et C. GUILLEMOT, « Oriented wavelet transform for image compression and denoising », t. 15, n° 10, p. 2892-2903, 2006 (cf. p. 60).

[Chen, 2015] C. CHEN, A. SEFF, A. KORNHAUSER et J. XIAO, « Deepdriving : Learning affordance for direct perception in autonomous driving », *Proceedings of the IEEE international conference on computer vision*, 2015, p. 2722-2730 (cf. p. 1, 94).

[Chen, 2020a] H. CHEN, W. XIE, A. VEDALDI et A. ZISSERMAN, « Vggsound : A large-scale audio-visual dataset », *ICASSP*, 2020 (cf. p. 76, 83).

[Chen, 2019] K. CHEN, J. WANG, J. PANG, Y. CAO, Y. XIONG, X. LI, S. SUN, W. FENG, Z. LIU, J. XU, Z. ZHANG, D. CHENG, C. ZHU, T. CHENG, Q. ZHAO, B. LI, X. LU, R. ZHU, Y. WU, J. DAI, J. WANG, J. SHI, W. OUYANG, C. C. LOY et D. LIN, « MMDetection : Open MMLab Detection Toolbox and Benchmark », *arXiv preprint arXiv:1906.07155*, 2019 (cf. p. 33).

[Chen, 2014] L.-C. CHEN, G. PAPANDREOU, I. KOKKINOS, K. MURPHY et A. L. YUILLE, « Semantic image segmentation with deep convolutional nets and fully connected crfs », *arXiv preprint arXiv:1412.7062*, 2014 (cf. p. 37).

[Chen, 2020b] M. CHEN, A. RADFORD, R. CHILD, J. WU, H. JUN, D. LUAN et I. SUTSKEVER, « Generative pretraining from pixels », *International Conference on Machine Learning*, 2020 (cf. p. 39).

[Chen, 2020c] T. CHEN, S. KORNBLITH, M. NOROUZI et G. HINTON, « A simple framework for contrastive learning of visual representations », *arXiv preprint arXiv:2002.05709*, 2020 (cf. p. 4, 14, 37, 40, 45, 49, 86, 97).

[Chen, 2021a] X. CHEN et K. HE, « Exploring simple siamese representation learning », *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2021, p. 15750-15758 (cf. p. 13, 14, 40).

[Chen, 2021b] X. CHEN, S. XIE et K. HE, « An empirical study of training self-supervised vision transformers », *arXiv preprint arXiv:2104.02057*, 2021 (cf. p. 4, 48, 49, 52, 97).

[Cheng, 2020] Z. CHENG, H. SUN, M. TAKEUCHI et J. KATTO, « Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules », *Computer Vision and Pattern Recognition*, 2020 (cf. p. 56, 61, 64).

[Child, 2019] R. CHILD, S. GRAY, A. RADFORD et I. SUTSKEVER, « Generating long sequences with sparse transformers », *arXiv preprint arXiv:1904.10509*, 2019 (cf. p. 9, 19).

[Choromanski, 2020] K. CHOROMANSKI, V. LIKHOSHERSTOV, D. DOHAN, X. SONG, A. GANE, T. SARLOS, P. HAWKINS, J. DAVIS, A. MOHIUDDIN, L. KAISER et al., « Rethinking attention with performers », *arXiv preprint arXiv:2009.14794*, 2020 (cf. p. 10, 19).

[Chum, 2007] O. CHUM, J. PHILBIN, J. SIVIC, M. ISARD et A. ZISSERMAN, « Total recall : Automatic query expansion with a generative feature model for object retrieval », *International Conference on Computer Vision*, 2007 (cf. p. 101).

[Contributors, 2020] M. CONTRIBUTORS, *MMSegmentation : OpenMMLab Semantic Segmentation Toolbox and Benchmark*, https://github.com/open-mmlab/mmsegmentation, 2020 (cf. p. 34).

[Cover, 1991] T. M. COVER, *Elements of information theory*. John Wiley & Sons, 1991 (cf. p. 15, 60).

[Cubuk, 2020] E. D. CUBUK, B. ZOPH, J. SHLENS et Q. V. LE, « Randaugment : Practical automated data augmentation with a reduced search space », *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops*, 2020 (cf. p. 28, 86).
[dAscoli, 2021] S. D'ASCOLI, H. TOUVRON, M. LEAVITT, A. MORCOS, G. BIROLI et L. SAGUN, « ConViT : Improving Vision Transformers with Soft Convolutional Inductive Biases », *arXiv preprint arXiv:2103.10697*, 2021 (cf. p. 19).

[Dai, 2019] Z. DAI, Z. YANG, Y. YANG, J. CARBONELL, Q. V. LE et R. SALAKHUTDINOV, « Transformer-xl : Attentive language models beyond a fixed-length context », *arXiv preprint arXiv:1901.02860*, 2019 (cf. p. 10).

[Deng, 2009a] J. DENG, W. DONG, R. SOCHER, L.-J. LI, K. LI et L. FEI-FEI, « ImageNet : A Large-Scale Hierarchical Image Database », *Computer Vision and Pattern Recognition*, 2009 (cf. p. 63, 64).

[Deng, 2009b] J. DENG, W. DONG, R. SOCHER, L.-J. LI, K. LI et L. FEI-FEI, « Imagenet : A large-scale hierar- chical image database », *Computer Vision and Pattern Recognition*, 2009 (cf. p. 4, 28, 37, 46, 97).

[Devlin, 2018] J. DEVLIN, M.-W. CHANG, K. LEE et K. TOUTANOVA, « Bert : Pre-training of deep bidirectional transformers for language understanding », *arXiv preprint arXiv:1810.04805*, 2018 (cf. p. 3-5, 9, 12, 38, 39, 90, 96-98).

[Ding, 2021a] M. DING, Z. YANG, W. HONG, W. ZHENG, C. ZHOU, D. YIN, J. LIN, X. ZOU, Z. SHAO, H. YANG et J. TANG, « CogView : Mastering Text-to-Image Generation via Transformers », *Advances in Neural Information Processing Systems*, 2021 (cf. p. 56, 60).

[Ding, 2021b] X. DING, X. ZHANG, J. HAN et G. DING, « RepMLP : Re-parameterizing Convolutions into Fully- connected Layers for Image Recognition », *arXiv preprint arXiv:2105.01883*, 2021 (cf. p. 20).

[Doersch, 2015] C. DOERSCH, A. GUPTA et A. A. EFROS, « Unsupervised visual representation learning by context prediction », *International Conference on Computer Vision*, 2015 (cf. p. 13).

[Doersch, 2020] C. DOERSCH, A. GUPTA et A. ZISSERMAN, « Crosstransformers : spatially-aware few-shot trans- fer », *arXiv preprint arXiv:2007.11498*, 2020 (cf. p. 37).

[Dong, 2015] C. DONG, C. C. LOY, K. HE et X. TANG, « Image super-resolution using deep convolutional networks », t. 38, n° 2, p. 295-307, 2015 (cf. p. 54).

[Doosivitskiy, 2021a] A. DOSOVITSKIY, L. BEYER, A. KOLESNIKOV, D. WEISSENBORN, X. ZHAI, T. UNTERTHINER, M. DEHGHANI, M. MINDERER, G. HEIGOLD, S. GELLY et al., « An image is worth 16x16 words : Transformers for image recognition at scale », *International Conference on Learning Representations*, 2021 (cf. p. 2, 3, 5, 11, 17, 24-27, 29, 31, 37, 40, 51, 75, 89, 95, 97, 98).

[Doosivitskiy, 2021b] A. DOSOVITSKIY, L. BEYER, A. KOLESNIKOV, D. WEISSENBORN, X. ZHAI, T. UNTERTHINER, M. DEHGHANI, M. MINDERER, G. HEIGOLD, S. GELLY, J. USZKOREIT et N. HOULSBY, « An Image is Worth 16x16 Words : Transformers for Image Recognition at Scale », *International Conference on Learning Representations*, 2021 (cf. p. 59).

[Doosivitskiy, 2015] A. DOSOVITSKIY, P. FISCHER, J. T. SPRINGENBERG, M. RIEDMILLER et T. BROX, « Discriminative unsupervised feature learning with exemplar convolutional neural networks », *IEEE transactions on pattern analysis and machine intelligence*, t. 38, n° 9, p. 1734-1747, 2015 (cf. p. 40).

[Doosivitskiy, 2014] A. DOSOVITSKIY, J. T. SPRINGENBERG, M. RIEDMILLER et T. BROX, « Discriminative un- supervised feature learning with convolutional neural networks », *Advances in neural information processing systems*, t. 27, p. 766-774, 2014 (cf. p. 13, 40).

[Dupont, 2021] E. DUPONT, A. GOLIŃSKI, M. ALIZADEH, Y. W. TEH et A. DOUCET, « COIN : COmpression with Implicit Neural representations », *ICLR Neural Compression Workshop*, 2021 (cf. p. 56).

[Dupont, 2022] E. DUPONT, H. LOYA, M. ALIZADEH, A. GOLIŃKI, Y. W. TEH et A. DOUCET, « COIN++ : Data Agnostic Neural Compression », t. arXiv:2201.12904, 2022 (cf. p. 56).

[Esser, 2021] P. ESSER, R. ROMBACH et B. OMMER, « Taming Transformers for High-Resolution Image Synthe- sis », *Computer Vision and Pattern Recognition*, 2021 (cf. p. 54, 56, 58, 60, 65).

[Faghri, 2017] F. FAGHRI, D. J. FLEET, J. R. KIROS et S. FIDLER, « Vse++ : Improving visual-semantic embed- dings with hard negatives », *arXiv preprint arXiv:1707.05612*, 2017 (cf. p. 72).

[Fan, 2021] H. FAN, B. XIONG, K. MANGALAM, Y. LI, Z. YAN, J. MALIK et C. FEICHTENHOFER, « Multiscale Vision Transformers », *arXiv preprint arXiv:2104.11227*, 2021 (cf. p. 11, 17, 20).

[Fang, 2021] H. FANG, P. XIONG, L. XU et Y. CHEN, « Clip2video : Mastering video-text retrieval via image clip », *arXiv preprint arXiv:2106.11097*, 2021 (cf. p. 72).

[Font, 2013] F. FONT, G. ROMA et X. SERRA, « Freesound technical demo », *ACM international conference on Multimedia*, 2013 (cf. p. 76, 83).

[Frey, 1998] B. J. FREY, « Bayesian networks for pattern classification, data compression, and channel coding. », thèse de doct., University of Toronto, 1998 (cf. p. 56).
References

[Frome, 2013] A. FROME, G. S. CORRADO, J. SHLENS, S. BENGIO, J. DEAN, M. RANZATO et T. MIKOLOV, « Devise : A deep visual-semantic embedding model », *Advances in neural information processing systems*, t. 26, 2013 (cf. p. 72).

[Fukushima, 1980] K. FUKUSHIMA, « Neocognitron : A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position », *Biological cybernetics*, t. 36, n° 4, p. 193-202, 1980 (cf. p. 1, 94).

[Gafni, 2022] O. GAFNI, A. POLYAK, O. ASHUAL, S. SHEYNIN, D. PARIKH et Y. TAIGMAN, « Make-A-Scene : Scene-Based Text-to-Image Generation with Human Priors », *European Conference on Computer Vision*, 2022 (cf. p. 56, 60).

[Gedas Bertasius, 2021] L. T. GEDAS BERTASIUS Heng Wang, « Is Space-Time Attention All You Need for Video Understanding? », 2021 (cf. p. 5, 98).

[Geirhos, 2018] R. GEIRHOS, P. RUBISCH, C. MICHAELIS, M. BETHGE, F. A. WICHMANN et W. BRENDEL, « ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness », *arXiv preprint arXiv:1811.12231*, 2018 (cf. p. 1).

[Gemmeke, 2017] J. F. GEMMEKE, D. P. W. ELLIS, D. FREEDMAN, A. JANSEN, W. LAWRENCE, R. C. MOORE, M. PLAKAL et M. RITTER, « Audio Set : An ontology and human-labeled dataset for audio events », *International Conference on Acoustics, Speech, and Signal Processing*, 2017 (cf. p. 76, 82).

[Gidaris, 2018] S. GIDARIS, P. SINGH et N. KOMODAKIS, « Unsupervised representation learning by predicting image rotations », *arXiv preprint arXiv:1803.07728*, 2018 (cf. p. 13).

[Girdhar, 2022a] R. GIRDHAR, A. EL-NOUBY, M. SINGH, K. V. ALWALA, A. JOULIN et I. MISRA, « OmniMAE : Single Model Masked Pretraining on Images and Videos », *arXiv preprint arXiv:2206.08356*, 2022 (cf. p. 72, 75).

[Girdhar, 2022b] R. GIRDHAR, M. SINGH, N. RAVI, L. van der MAATEN, A. JOULIN et I. MISRA, « Omnivore : A Single Model for Many Visual Modalities », *Computer Vision and Pattern Recognition*, 2022 (cf. p. 72, 75, 76, 83).

[Gong, 2021] Y. GONG, Y.-A. CHUNG et J. GLASS, « AST : Audio Spectrogram Transformer », *Interspeech*, 2021 (cf. p. 75, 77).

[Goodfellow, 2014] I. GOODFELLOW, J. POUGET-ABADIE, M. MIRZA, B. XU, D. WARDE-FARLEY, S. OZAIR, A. COURVILLE et Y. BENGIO, « Generative Adversarial Nets », *Advances in Neural Information Processing Systems*, 2014 (cf. p. 56, 62).

[Gordo, 2017] A. GORDO, J. ALMAZÁN, J. REVAUD et D. LARLUS, « End-to-End Learning of Deep Visual Representations for Image Retrieval », *International journal of Computer Vision*, t. 124, 2017 (cf. p. 100).

[Goyal, 2021] P. GOYAL, M. CARON, B. LEFAUDEUX, M. XU, P. WANG, V. PAI, M. SINGH, V. LIPTCHINSKY, I. MISRA, A. JOULIN et al., « Self-supervised pretraining of visual features in the wild », *arXiv preprint arXiv:2103.01988*, 2021 (cf. p. 38, 40).

[Graham, 2021] B. GRAHAM, A. EL-NOUBY, H. TOUVRON, P. STOCK, A. JOULIN, H. JÉGOU et M. DOUZE, « LeViT : a Vision Transformer in ConvNet's Clothing for Faster Inference », *arXiv preprint arXiv:2104.01136*, 2021 (cf. p. 11, 19, 25).

[Grauman, 2022] K. GRAUMAN, A. WESTBURY, E. BYRNE, Z. CHAVIS, A. FURNARI, R. GIRDHAR, J. HAMBURGER, H. JIANG, M. LIU, X. LIU et al., « Ego4d : Around the world in 3,000 hours of egocentric video », *Computer Vision and Pattern Recognition*, 2022 (cf. p. 76, 84).

[Gray, 1998] R. M. GRAY et D. L. NEUHOFF, « Quantization », t. 44, n° 6, p. 2325-2383, 1998 (cf. p. 54, 60).

[Gregor, 2016] K. GREGOR, F. BESSE, D. JIMENEZ REZENDE, I. DANIELKLA et D. WIERSTRA, « Towards conceptual compression », *Advances in Neural Information Processing Systems*, 2016 (cf. p. 56).

[Grill, 2020] J.-B. GRILL, F. STRUB, F. ALTCHÉ, C. TALLEC, P. H. RICHEMOND, E. BUCHATSKAYA, C. DOERSCH, B. A. PIRES, Z. D. GUO, M. G. AZAR et al., « Bootstrap your own latent : A new approach to self-supervised learning », *arXiv preprint arXiv:2006.07733*, 2020 (cf. p. 14, 37, 40).

[Gu, 2021] X. GU, T.-Y. LIN, W. KUO et Y. CUI, « Open-vocabulary object detection via vision and language knowledge distillation », *arXiv preprint arXiv:2104.13921*, 2021 (cf. p. 73).

[Gulati, 2020] A. GULATI, J. QIN, C.-C. CHIU, N. PARMAR, Y. ZHANG, J. YU, W. HAN, S. WANG, Z. ZHANG, Y. WU et al., « Conformer : Convolution-augmented transformer for speech recognition », *arXiv preprint arXiv:2005.08100*, 2020 (cf. p. 9, 10).
[Guzhov, 2021] A. GUZHOV, F. RAUE, J. HEES et A. DENGEL, *AudioCLIP: Extending CLIP to Image, Text and Audio*, 2021. arXiv : 2106.13043 [cs.SD] (cf. p. 73, 74, 76-78).

[Hadsell, 2006] R. HADSELL, S. CHOPRA et Y. LECUN, « Dimensionality reduction by learning an invariant mapping », *Computer Vision and Pattern Recognition*, 2006 (cf. p. 73).

[Han, 2021] K. HAN, A. XIAO, E. WU, J. GUO, C. XU et Y. WANG, « Transformer in transformer », *arXiv preprint arXiv:2103.00112*, 2021 (cf. p. 19).

[He, 2021a] K. HE, X. CHEN, S. XIE, Y. LI, P. DOLLAR et R. GIRSHICK, « Masked Autoencoders Are Scalable Vision Learners », *Computer Vision and Pattern Recognition*, 2021 (cf. p. 54, 90).

[He, 2021b] K. HE, X. CHEN, S. XIE, Y. LI, P. DOLLÁR et R. GIRSHICK, « Masked Autoencoders Are Scalable Vision Learners », *arXiv preprint arXiv:2111.06377*, 2021 (cf. p. 44, 52, 92).

[He, 2020] K. HE, H. FAN, Y. WU, S. XIE et R. GIRSHICK, « Momentum contrast for unsupervised visual representation learning », *Computer Vision and Pattern Recognition*, 2020 (cf. p. 4, 14, 37, 40, 97).

[He, 2018] K. HE, R. GIRSHICK et P. DOLLÁR, « Rethinking im-agenet pre-training », *arXiv preprint arXiv:1811.08883*, 2018 (cf. p. 46).

[He, 2017] K. HE, G. GKIOXARI, P. DOLLÁR et R. GIRSHICK, « Mask r-cnn », *International Conference on Computer Vision*, 2017 (cf. p. 32, 33, 37, 47, 52).

[He, 2016] K. HE, X. ZHANG, S. REN et J. SUN, « Deep residual learning for image recognition », *Computer Vision and Pattern Recognition*, 2016 (cf. p. 1, 7, 17, 31, 33, 34, 37, 94).

[Heusel, 2017] M. HEUSEL, H. RAMSAUER, T. UNTERTHINER, B. NESSLER et S. HOCHREITER, « GANs trained by a two time-scale update rule converge to a local Nash equilibrium », *Advances in Neural Information Processing Systems*, 2017 (cf. p. 64).

[Hinton, 2006] G. E. HINTON et R. R. SALAKHUTDINOV, « Reducing the dimensionality of data with neural networks », *science*, t. 313, n° 5786, p. 504-507, 2006 (cf. p. 39).

[Ho, 2019] J. HO, N. KALCHBRENNER, D. WEISSENBORN et T. SALIMANS, « Axial attention in multidimensional transformers », *arXiv preprint arXiv:1912.12180*, 2019 (cf. p. 19, 27).

[Horn, 2017] G. V. HORN, O. MAC AODHA, Y. SONG, A. SHEPARD, H. ADAM, P. PERONA et S. J. BELONGIE, « The iNaturalist Species Classification and Detection Dataset », *arXiv preprint arXiv:1707.06642*, 2017 (cf. p. 29, 37, 46).

[Howard, 2017] A. G. HOWARD, M. ZHU, B. CHEN, D. KALENICHENKO, W. WANG, T. WEYAND, M. ANDREETTO et H. ADAM, « Mobilenets : Efficient convolutional neural networks for mobile vision applications », *arXiv preprint arXiv:1704.04861*, 2017 (cf. p. 8).

[Hu, 2018] J. HU, L. SHEN et G. SUN, « Squeeze-and-excitation networks », *Computer Vision and Pattern Recognition*, 2018 (cf. p. 1, 11, 20, 94).

[Huang, 2016] G. HUANG, Y. SUN, Z. LIU, D. SEDRA et K. Q. WEINBERGER, « Deep networks with stochastic depth », *European Conference on Computer Vision*, 2016 (cf. p. 28, 82).

[Ilharco, 2021] G. ILHARCO, M. WORTSMAN, R. WIGHTMAN, C. GORDON, N. CARLINI, R. TAORI, A. DAVE, V. SHANKAR, H. NAMKOONG, J. MILLER, H. HAJISHIRZI, A. FARHADI et L. SCHMIDT, *OpenCLIP*, 2021 (cf. p. 15, 75, 76).

[Jaegle, 2021] A. JAEGLE, F. GIMENO, A. BROCK, A. ZISSERMAN, O. VINYALS et J. CARREIRA, « Perceiver : General Perception with Iterative Attention », *arXiv preprint arXiv:2103.03206*, 2021 (cf. p. 19).

[Jégou, 2010] H. JÉGOU, C. SCHMID, H. HARZALLAH et J. VERBEEK, « Accurate image search using the contextual dissimilarity measure », t. 32, n° 1, p. 2-11, 2010 (cf. p. 54, 58).

[Jégou, 2008] H. JÉGOU, M. DOUZE et C. SCHMID, « Hamming embedding and weak geometric consistency for large scale image search », *European Conference on Computer Vision*, 2008 (cf. p. 101).

[Jégou, 2012] H. JÉGOU, F. PERRONNIN, M. DOUZE, J. SÁNCHEZ, P. PEREZ et C. SCHMID, « Aggregating local image descriptors into compact codes », *IEEE Transactions on Pattern Analysis and Machine Intelligence*, t. 34, n° 9, 2012 (cf. p. 100, 101).

[Jia, 2021a] C. JIA, Y. YANG, Y. XIA, Y.-T. CHEN, Z. PAREKH, H. PHAM, Q. LE, Y.-H. SUNG, Z. LI et T. DUERIG, « Scaling up visual and vision-language representation learning with noisy text supervision », *International Conference on Machine Learning*, 2021 (cf. p. 15, 71, 72).

[Jia, 2021b] X. JIA, C. ZHU, M. LI, W. TANG et W. ZHOU, « LLVIP : A visible-infrared paired dataset for low-light vision », *International Conference on Computer Vision*, 2021 (cf. p. 76, 84).
References

[Johnson, 2017] M. JOHNSON, M. SCHUSTER, Q. V. LE, M. KRIKUN, Y. WU, Z. CHEN, N. THORAT, F. VIÉGAS, M. WATTENBERG, G. CORRADO et al., « Google's multilingual neural machine translation system : Enabling zero-shot translation », *Transactions of the Association for Computational Linguistics*, t. 5, p. 339-351, 2017 (cf. p. 73).

[Joulin, 2016] A. JOULIN, L. VAN DER MAATEN, A. JABRI et N. VASILACHE, « Learning visual features from large weakly supervised data », *European Conference on Computer Vision*, Springer, 2016, p. 67-84 (cf. p. 5, 15, 40, 72, 98).

[Karras, 2019] T. KARRAS, S. LAINE et T. AILA, « A Style-Based Generator Architecture for Generative Adversarial Networks », *Computer Vision and Pattern Recognition*, 2019 (cf. p. 65).

[Kataoka, 2020] H. KATAOKA, K. OKAYASU, A. MATSUMOTO, E. YAMAGATA, R. YAMADA, N. INOUE, A. NAKAMURA et Y. SATOH, « Pre-training without natural images », *Proceedings of the Asian Conference on Computer Vision*, 2020 (cf. p. 40).

[Katharopoulos, 2020] A. KATHAROPOULOS, A. VYAS, N. PAPPAS et F. FLEURET, « Transformers are RNNs : Fast autoregressive transformers with linear attention », *International Conference on Machine Learning*, 2020 (cf. p. 19).

[Kay, 2017] W. KAY, J. CARREIRA, K. SIMONYAN, B. ZHANG, C. HILLIER, S. VIJAYANARASIMHAN, F. VIOLA, T. GREEN, T. BACK, P. NATSEV, A. SULEYMAN et A. ZISSERMAN, « The kinetics human action video dataset », *arXiv preprint arXiv:1705.06950*, 2017 (cf. p. 77).

[Kim, 2019] C. D. KIM, B. KIM, H. LEE et G. KIM, « Audiocaps : Generating captions for audios in the wild », *NAACL*, 2019 (cf. p. 76, 83).

[Kingma, 2014] D. KINGMA et M. WELLING, « Auto-Encoding Variational Bayes », *International Conference on Learning Representations*, 2014 (cf. p. 56).

[Kirillov, 2019] A. KIRILLOV, R. GIRSHICK, K. HE et P. DOLLÁR, « Panoptic feature pyramid networks », *Computer Vision and Pattern Recognition*, 2019 (cf. p. 34).

[Kiros, 2014] R. KIROS, R. SALAKHUTDINOV et R. S. ZEMEL, « Unifying visual-semantic embeddings with multimodal neural language models », *arXiv preprint arXiv:1411.2539*, 2014 (cf. p. 72).

[Kitaev, 2020] N. KITAEV, Ł. KAISER et A. LEVSKAYA, « Reformer : The efficient transformer », *arXiv preprint arXiv:2001.04451*, 2020 (cf. p. 10).

[Knoll, 2020] F. KNOLL, K. HAMMERNIK, C. ZHANG, S. MOELLER, T. POCK, D. K. SODICKSON et M. AKCAKAYA, « Deep-learning methods for parallel magnetic resonance imaging reconstruction : A survey of the current approaches, trends, and issues », *IEEE Signal Processing Magazine*, t. 37, n° 1, p. 128-140, 2020 (cf. p. 54).

[Kodak, 1993] E. KODAK, Kodak lossless true color image suite (PhotoCD PCD0992), 1993 (cf. p. 64).

[Krause, 2013] J. KRAUSE, M. STARK, J. DENG et L. FEI-FEI, « 3D Object Representations for Fine-Grained Categorization », *4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13)*, 2013 (cf. p. 29, 45, 46).

[Krishna, 2021] K. KRISHNA, J. BIGHAM et Z. C. LIPTON, « Does Pretraining for Summarization Require Knowledge Transfer ? », *arXiv preprint arXiv:2109.04953*, 2021 (cf. p. 40).

[Krizhevsky, 2009] A. KRIZHEVSKY, « Learning Multiple Layers of Features from Tiny Images », CIFAR, rapp. tech., 2009 (cf. p. 29).

[Krizhevsky, 2012] A. KRIZHEVSKY, I. SUTSKEVER et G. E. HINTON, « Imagenet classification with deep convolutional neural networks », *Advances in Neural Information Processing Systems*, 2012 (cf. p. 7).

[Lample, 2017] G. LAMPLE, A. CONNEAU, L. DENOYER et M. RANZATO, « Unsupervised machine translation using monolingual corpora only », *arXiv preprint arXiv:1711.00043*, 2017 (cf. p. 73).

[Larochelle, 2011] H. LAROCHELLE et I. MURRAY, « The Neural Autoregressive Distribution Estimator », 2011 (cf. p. 56).

[Larsson, 2016] G. LARSSON, M. MAIRE et G. SHAKHNAROVICH, « Learning representations for automatic colorization », *Computer Vision-ECCV 2016 : 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14*, Springer, 2016, p. 577-593 (cf. p. 13).

[LeCun, 1989] Y. LECUN, B. BOSER, J. S. DENKER, D. HENDERSON, R. E. HOWARD, W. HUBBARD et L. D. JACKEL, « Backpropagation applied to handwritten zip code recognition », *Neural computation*, t. 1, n° 4, p. 541-551, 1989 (cf. p. 1, 7, 94).

[LeCun, 1998] Y. LECUN, L. BOTTOU, Y. BENGIO et P. HAFFNER, « Gradient-based learning applied to document recognition », *Proceedings of the IEEE*, 1998 (cf. p. 7).
[Ledig, 2017] C. LEDIG, L. THEIS, F. HUSZAR, J. CABALLERO, A. CUNNINGHAM, A. ACOSTA, A. AITKEN, A. TEJANI, J. TOTZ, Z. WANG et W. SHI, « Photo-realistic single image super-resolution using a generative adversarial network », *Computer Vision and Pattern Recognition*, 2017 (cf. p. 54).

[Lee-Thorp, 2021] J. LEE-THORP, J. AINSLIE, I. ECKSTEIN et S. ONTANON, « FNet : Mixing Tokens with Fourier Transforms », *arXiv preprint arXiv:2105.03824*, 2021 (cf. p. 19).

[Lewis, 2019] M. LEWIS, Y. LIU, N. GOYAL, M. GHAZVININEJAD, A. MOHAMED, O. LEVY, V. STOYANOV et L. ZETTLEMOYER, « Bart : Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension », *arXiv preprint arXiv:1910.13461*, 2019 (cf. p. 39).

[Lezama, 2022] J. LEZAMA, H. CHANG, L. JIANG et I. ESSA, « Improved Masked Image Generation with Token-Critic », *European Conference on Computer Vision*, 2022 (cf. p. 57).

[Li, 2022] B. LI, K. Q. WEINBERGER, S. BELONGIE, V. KOLTUN et R. RANFTL, « Language-driven semantic segmentation », *arXiv preprint arXiv:2201.03546*, 2022 (cf. p. 73).

[Likhosherstov, 2021] V. LIKHOSHERSTOV, A. ARNAB, K. CHOROMANSKI, M. LUCIC, Y. TAY, A. WELLER et M. DEHGHANI, « PolyViT : Co-training Vision Transformers on Images, Videos and Audio », *arXiv preprint arXiv:2111.12993*, 2021 (cf. p. 72).

[Lin, 2017] T.-Y. LIN, P. DOLLÁR, R. GIRSHICK, K. HE, B. HARIHARAN et S. BELONGIE, « Feature pyramid networks for object detection », *Computer Vision and Pattern Recognition*, 2017 (cf. p. 32, 52).

[Lin, 2014] T.-Y. LIN, M. MAIRE, S. BELONGIE, J. HAYS, P. PERONA, D. RAMANAN, P. DOLLÁR et C. L. ZITNICK, « Microsoft coco : Common objects in context », *European Conference on Computer Vision*, 2014 (cf. p. 32, 45, 46).

[Lin, 2022] Z. LIN, S. GENG, R. ZHANG, P. GAO, G. de MELO, X. WANG, J. DAI, Y. QIAO et H. LI, « Frozen CLIP Models are Efficient Video Learners », *European Conference on Computer Vision*, 2022 (cf. p. 72).

[Liu, 2022a] X. LIU, J. ZHOU, T. KONG, X. LIN et R. JI, « Exploring target representations for masked autoencoders », *arXiv preprint arXiv:2209.03917*, 2022 (cf. p. 73).

[Liu, 2021a] Y. LIU, E. SANGINETO, W. BI, N. SEBE, B. LEPRI et M. NADAI, « Efficient Training of Visual Transformers with Small Datasets », *Advances in Neural Information Processing Systems*, 2021 (cf. p. 47, 48).

[Liu, 2021b] Z. LIU, Y. LIN, Y. CAO, H. HU, Y. WEI, Z. ZHANG, S. LIN et B. GUO, « Swin transformer : Hierarchical vision transformer using shifted windows », *arXiv preprint arXiv:2103.14030*, 2021 (cf. p. 11, 17, 18, 20, 28, 29, 31, 33, 34, 37, 52).

[Liu, 2022b] Z. LIU, H. MAO, C.-Y. WU, C. FEICHTENHOFER, T. DARRELL et S. XIE, « A convnet for the 2020s », *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2022, p. 11976-11986 (cf. p. 12).

[Loshchilov, 2017] I. LOSCHCHLOV et F. HUTTER, « Decoupled weight decay regularization », *arXiv preprint arXiv:1711.05101*, 2017 (cf. p. 28, 63).

[Lu, 2019] X. LU, H. WANG, W. DONG, F. WU, Z. ZHENG et G. SHI, « Learning a deep vector quantization network for image compression », *IEEE Access*, t. 7, p. 118815-118825, 2019 (cf. p. 56).

[Ludwig, 2016] R. LUDWIG et J. TAYLOR, *Voyager telecommunications*. John Wiley et Sons, Inc, 2016 (cf. p. 54).

[Luo, 2021] H. LUO, L. JI, M. ZHONG, Y. CHEN, W. LEI, N. DUAN et T. LI, « CLIP4Clip : An Empirical Study of CLIP for End to End Video Clip Retrieval. CoRR abs/2104.08860 (2021) », *arXiv preprint arXiv:2104.08860*, 2021 (cf. p. 72).

[MacKay, 2003] D. J. C. MacKAY, *Information Theory, Inference, and Learning Algorithms*. Cambridge University Press, 2003 (cf. p. 15).

[Mahajan, 2018] D. MAHAJAN, R. GIRSHICK, V. RAMANATHAN, K. HE, M. PALURI, Y. LI, A. BHARAMBE et L. VAN DER MAATEN, « Exploring the limits of weakly supervised pretraining », *Proceedings of the European conference on computer vision (ECCV)*, 2018, p. 181-196 (cf. p. 40, 71).

[Melas-Kyriazi, 2021] L. MELAS-KYRIAZI, « Do You Even Need Attention ? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet », *arXiv preprint arXiv:2105.02723*, 2021 (cf. p. 20).

[Mentzer, 2018] F. MENTZER, E. AGUSTSSON, M. TSCHANNEN, R. TIMOFTE et L. van GOOL, « Conditional probability models for deep image compression », *Computer Vision and Pattern Recognition*, 2018 (cf. p. 56).

[Mentzer, 2020] F. MENTZER, G. D. TODERICI, M. TSCHANNEN et E. AGUSTSSON, « High-Fidelity Generative Image Compression », *Advances in Neural Information Processing Systems*, 2020 (cf. p. 54-56, 63-65).
[Miech, 2020] A. MIECH, J.-B. ALAYRAC, L. SMAIRA, I. LAPTEV, J. SIVIC et A. ZISSERMAN, « End-to-end learning of visual representations from uncurated instructional videos », *Computer Vision and Pattern Recognition*, 2020 (cf. p. 72).

[Miech, 2019a] A. MIECH, D. ZHUKOV, J.-B. ALAYRAC, M. TAPASWI, I. LAPTEV et J. SIVIC, « HowTo100M : Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips », *International Conference on Computer Vision*, 2019 (cf. p. 78).

[Miech, 2019b] A. MIECH, D. ZHUKOV, J.-B. ALAYRAC, M. TAPASWI, I. LAPTEV et J. SIVIC, « Howto100m : Learning a text-video embedding by watching hundred million narrated video clips », *International Conference on Computer Vision*, 2019 (cf. p. 72).

[Minnen, 2018] D. MINNEN, J. BALLÉ et G. D. TODERICI, « Joint Autoregressive and Hierarchical Priors for Learned Image Compression », *Advances in Neural Information Processing Systems*, 2018 (cf. p. 56, 61, 64, 65).

[Minnen, 2020] D. MINNEN et S. SINGH, « Channel-wise autoregressive entropy models for learned image compression », 2020 (cf. p. 56).

[Morgado, 2021] P. MORGADO, N. VASCONCELOS et I. MISRA, « Audio-Visual Instance Discrimination with Cross-Modal Agreement », *Computer Vision and Pattern Recognition*, 2021 (cf. p. 71-73, 79).

[Nagrani, 2022] A. NAGRANI, P. H. SEO, B. SEYBOLD, A. HAUTH, S. MANEN, C. SUN et C. SCHMID, « Learning Audio-Video Modalities from Image Captions », *European Conference on Computer Vision*, 2022 (cf. p. 72, 77, 78).

[Nagrani, 2021] A. NAGRANI, S. YANG, A. ARNAB, A. JANSEN, C. SCHMID et C. SUN, « Attention bottlenecks for multimodal fusion », *Advances in Neural Information Processing Systems*, 2021 (cf. p. 78).

[Naseer, 2021] M. M. NASEER, K. RANASINGHE, S. H. KHAN, M. HAYAT, F. SHAHBAZ KHAN et M.-H. YANG, « Intriguing properties of vision transformers », *Advances in Neural Information Processing Systems*, t. 34, p. 23 296-23 308, 2021 (cf. p. 2, 95).

[Nilsback, 2008] M.-E. NILSBACK et A. ZISSERMAN, « Automated Flower Classification over a Large Number of Classes », *Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing*, 2008 (cf. p. 29).

[Noroozi, 2016] M. NOROOZI et P. FAVARO, « Unsupervised learning of visual representations by solving jigsaw puzzles », *European conference on computer vision*, Springer, 2016, p. 69-84 (cf. p. 12, 39).

[El-Nouby, 2021a] A. EL-NOUBY, G. IZACARD, H. TOUVRON, I. LAPTEV, H. JEGOU et E. GRAVE, « Are Large-scale Datasets Necessary for Self-Supervised Pre-training ? », *arXiv preprint arXiv:2112.10740*, 2021 (cf. p. 4, 54, 90, 97).

[El-Nouby, 2023] A. EL-NOUBY, M. J. MUCKLEY, K. ULLRICH, I. LAPTEV, J. VERBEEK et H. JEGOU, « Image Compression with Product Quantized Masked Image Modeling », *Transactions on Machine Learning Research*, 2023 (cf. p. 4, 90, 98).

[El-Nouby, 2021b] A. EL-NOUBY, N. NEVEROVA, I. LAPTEV et H. JÉGOU, « Training vision transformers for image retrieval », *arXiv preprint arXiv:2102.05644*, 2021 (cf. p. 17, 89, 100, 101).

[El-Nouby, 2021c] A. EL-NOUBY, H. TOUVRON, M. CARON, P. BOJANOWSKI, M. DOUZE, A. JOULIN, I. LAPTEV, N. NEVEROVA, G. SYNNAEVE, J. VERBEEK et al., « XCIiT : Cross-Covariance Image Transformers », *Advances in Neural Information Processing Systems*, 2021 (cf. p. 3, 59, 89, 97).

[El-Nouby, 2021d] A. EL-NOUBY, H. TOUVRON, M. CARON, P. BOJANOWSKI, M. DOUZE, A. JOULIN, I. LAPTEV, N. NEVEROVA, G. SYNNAEVE, J. VERBEEK et al., « XCIiT : Cross-Covariance Image Transformers », *arXiv preprint arXiv:2106.09681*, 2021 (cf. p. 52).

[Oncescu, 2021] A.-M. ONCESCU, A. KOEPKE, J. F. HENRIQUES, Z. AKATA et S. ALBANIE, « Audio retrieval with natural language queries », *arXiv preprint arXiv:2105.02192*, 2021 (cf. p. 78, 83).

[Oord, 2016a] A. v. d. OORD, S. DIELEMAN, H. ZEN, K. SIMONYAN, O. VINYALS, A. GRAVES, N. KALCHBRENNER, A. SENIOR et K. KAVUKCUOGLU, « WaveNet : a generative model for raw audio », *ISCA Speech Syntesis Workshop*, 2016 (cf. p. 56).

[Oord, 2016b] A. v. d. OORD, N. KALCHBRENNER, O. VINYALS, L. ESPEHOLT, A. GRAVES et K. KAVUKCUOGLU, « Conditional Image Generation with PixelCNN Decoders », *Advances in Neural Information Processing Systems*, 2016 (cf. p. 54).

[Oord, 2017] A. v. d. OORD, O. VINYALS et K. KAVUKCUOGLU, « Neural Discrete Representation Learning », *Advances in Neural Information Processing Systems*, 2017 (cf. p. 54, 56, 58).
[Oord, 2018] A. v. d. OORD, Y. LI et O. VINYALS, « Representation learning with contrastive predictive coding », *arXiv preprint arXiv:1807.03748*, 2018 (cf. p. 13, 45, 74).

[Oquab, 2014] M. OQUAB, L. BOTTOU, I. LAPTEV et J. SIVIC, « Learning and transferring mid-level image representations using convolutional neural networks », *Computer Vision and Pattern Recognition*, 2014 (cf. p. 37).

[Owens, 2018] A. OWENS et A. A. EFROS, « Audio-visual scene analysis with self-supervised multisensory features », *European Conference on Computer Vision*, 2018 (cf. p. 71).

[Parmar, 2018] N. PARMAR, A. VASWANI, J. USZKOREIT, L. KAISER, N. SHAZEER, A. KU et D. TRAN, « Image transformer », *International Conference on Machine Learning*, 2018 (cf. p. 19).

[Pathak, 2016] D. PATHAK, P. KRAHENBUHL, J. DONAHUE, T. DARRELL et A. A. EFROS, « Context encoders : Feature learning by inpainting », *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2016, p. 2536-2544 (cf. p. 12, 39).

[Patrick, 2021] M. PATRICK, Y. M. ASANO, R. FONG, J. F. HENRIQUES, G. ZWEIG et A. VEDALDI, « Multi-modal self-supervision from generalized data transformations », *International Conference on Computer Vision*, 2021 (cf. p. 71).

[Patrick, 2020] M. PATRICK, P.-Y. HUANG, Y. ASANO, F. METZE, A. HAUPTMANN, J. HENRIQUES et A. VEDALDI, « Support-set bottlenecks for video-text representation learning », *arXiv preprint arXiv:2010.02824*, 2020 (cf. p. 78).

[Peng, 2019] X. PENG, Q. BAI, X. XIA, Z. HUANG, K. SAENKO et B. WANG, « Moment matching for multi-source domain adaptation », *International Conference on Computer Vision*, 2019 (cf. p. 45, 46).

[Peng, 2022] Z. PENG, L. DONG, H. BAO, Q. YE et F. WEI, « Beit v2 : Masked image modeling with vector-quantized visual tokenizers », *arXiv preprint arXiv:2208.06366*, 2022 (cf. p. 73, 92).

[Philbin, 2007] J. PHILBIN, O. CHUM, M. ISARD, J. SIVIC et A. ZISSERMAN, « Object Retrieval with Large Vocabularies and Fast Spatial Matching », *Computer Vision and Pattern Recognition*, 2007 (cf. p. 101).

[Piczak, 2015] K. J. PICZAK, « ESC : Dataset for environmental sound classification », 2015 (cf. p. 76, 83).

[Purushwalkam, 2020] S. PURUSHWALKAM et A. GUPTA, « Demystifying contrastive self-supervised learning : Invariances, augmentations and dataset biases », *arXiv preprint arXiv:2007.13916*, 2020 (cf. p. 38).

[Qiu, 2019] J. QIU, H. MA, O. LEVY, S. W.-t. YIH, S. WANG et J. TANG, « Blockwise self-attention for long document understanding », *arXiv preprint arXiv:1911.02972*, 2019 (cf. p. 19).

[Radenović, 2018a] F. RADENOVIĆ, A. ISCEN, G. TOLIAS, Y. AVRITHIS et O. CHUM, « Revisiting Oxford and Paris : Large-scale image retrieval benchmarking », *Computer Vision and Pattern Recognition*, 2018 (cf. p. 101).

[Radenović, 2018b] F. RADENOVIĆ, G. TOLIAS et O. CHUM, « Fine-tuning CNN image retrieval with no human annotation », *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2018 (cf. p. 102).

[Radford, 2021] A. RADFORD, J. W. KIM, C. HALLACY, A. RAMESH, G. GOH, S. AGARWAL, G. SASTRY, A. ASKELL, P. MISHKIN, J. CLARK et al., « Learning transferable visual models from natural language supervision », *International Conference on Machine Learning*, 2021 (cf. p. 5, 15, 71-73, 75-77, 79, 84, 85, 98).

[Radford, 2018] A. RADFORD, K. NARASIMHAN, T. SALIMANS et I. SUTSKEVER, « Improving language understanding with unsupervised learning », *Technical report, OpenAI*, 2018 (cf. p. 3-5, 9, 12, 96-98).

[Radosavovic, 2020] I. RADOSAVOVIC, R. P. KOSARAJU, R. GIRSHICK, K. HE et P. DOLLÁR, « Designing network design spaces », *Computer Vision and Pattern Recognition*, 2020 (cf. p. 8, 28, 29, 37).

[Raffel, 2019] C. RAFFEL, N. SHAZEER, A. ROBERTS, K. LEE, S. NARANG, M. MATENA, Y. ZHOU, W. LI et P. J. LIU, « Exploring the limits of transfer learning with a unified text-to-text transformer », *arXiv preprint arXiv:1910.10683*, 2019 (cf. p. 39, 40).

[Ramachandran, 2019] P. RAMACHANDRAN, N. PARMAR, A. VASWANI, I. BELLO, A. LEVSKAYA et J. SHLENS, « Stand-alone self-attention in vision models », *arXiv preprint arXiv:1906.05909*, 2019 (cf. p. 11).

[Ramesh, 2022a] A. RAMESH, P. DHARIWAL, A. NICHOL, C. CHU et M. CHEN, « Hierarchical Text-Conditional Image Generation with CLIP Latents », 2022 (cf. p. 1, 92, 94).

[Ramesh, 2022b] A. RAMESH, P. DHARIWAL, A. NICHOL, C. CHU et M. CHEN, « Hierarchical text-conditional image generation with clip latents », *arXiv preprint arXiv:2204.06125*, 2022 (cf. p. 71, 81).

[Ramesh, 2021a] A. RAMESH, M. PAVLOV, G. GOH, S. GRAY, C. VOSS, A. RADFORD, M. CHEN et I. SUTSKEVER, « Zero-Shot Text-to-Image Generation », *International Conference on Machine Learning*, 2021 (cf. p. 56, 60, 65).
References

[Ramesh, 2021b] A. RAMESH, M. PAVLOV, G. GOH, S. GRAY, C. VOSS, A. RADFORD, M. CHEN et I. SUTSKEVER, « Zero-shot text-to-image generation », *arXiv preprint arXiv:2102.12092*, 2021 (cf. p. 40).

[Ranftl, 2021] R. RANFTL, A. BOCHKOVSKIY et V. KOLTUN, « Vision Transformers for Dense Prediction », *arXiv preprint arXiv:2103.13413*, 2021 (cf. p. 20, 34).

[Ranzato, 2007] M. RANZATO, C. POULTNEY, S. CHOPRA, Y. LECUN et al., « Efficient learning of sparse representations with an energy-based model », *Advances in neural information processing systems*, t. 19, p. 1137, 2007 (cf. p. 39).

[Razavi, 2019] A. RAZAVI, A. van den OORD et O. VINYALS, « Generating Diverse High-Fidelity Images with VQ-VAE-2 », *Advances in Neural Information Processing Systems*, 2019 (cf. p. 54, 56, 58).

[Recht, 2019] B. RECHT, R. ROELOFS, L. SCHMIDT et V. SHANKAR, « Do imagenet classifiers generalize to imagenet ? », *International Conference on Machine Learning*, 2019 (cf. p. 28).

[Reed, 2017] S. REED, A. van den OORD, N. KALCHBRENNER, S. G. COLMENAREJO, Z. WANG, D. BELOV et N. de FREITAS, « Parallel Multiscale Autoregressive Density Estimation », *International Conference on Machine Learning*, 2017 (cf. p. 57, 60).

[Rezende, 2015] D. REZENDE et S. MOHAMED, « Variational Inference with Normalizing Flows », *International Conference on Machine Learning*, 2015 (cf. p. 56).

[Richardson, 2004] I. E. RICHARDSON, H. 264 and MPEG-4 video compression: video coding for next-generation multimedia. John Wiley & Sons, 2004 (cf. p. 59).

[Rifai, 2011] S. RIFAI, P. VINCENT, X. MULLER, X. GLOROT et Y. BENGIO, « Contractive auto-encoders: Explicit invariance during feature extraction », *Proceedings of the 28th international conference on international conference on machine learning*, 2011, p. 833-840 (cf. p. 12).

[Rippel, 2017] O. RIPPEL et L. BOURDEV, « Real-Time Adaptive Image Compression », *International Conference on Machine Learning*, 2017 (cf. p. 56).

[Rombach, 2021] R. ROMBACH, A. BLATTMANN, D. LORENZ, P. ESSER et B. OMMER, *High-Resolution Image Synthesis with Latent Diffusion Models*, 2021. arXiv : 2112.10752 [cs.CV] (cf. p. 54).

[Rosch, 1973] E. H. ROSCH, « Natural categories », *Cognitive Psychology*, 1973 (cf. p. 37).

[Russakovsky, 2015] O. RUSSAKOVSKY, J. DENG, H. SU, J. KRAUSE, S. SATHEESH, S. MA, Z. HUANG, A. KARPATHY, A. KHOSLA, M. BERNSTEIN, A. C. BERG et L. FEI-FEI, « ImageNet Large Scale Visual Recognition Challenge », *IJCV*, 2015 (cf. p. 77).

[Salimans, 2017] T. SALIMANS, A. KARPATHY, X. CHEN et D. KINGMA, « PixelCNN++: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications », *International Conference on Learning Representations*, 2017 (cf. p. 56).

[Santurkar, 2018] S. SANTURKAR, D. BUDDEN et N. SHAVIT, « Generative compression », *2018 Picture Coding Symposium (PCS)*, IEEE, 2018, p. 258-262 (cf. p. 56).

[Sauer, 2021] A. SAUER, K. CHITTA, J. MÜLLER et A. GEIGER, « Projected GANs Converge Faster », *Advances in Neural Information Processing Systems*, 2021 (cf. p. 63, 65).

[Schönfeld, 2020] E. SCHÖNFELD, B. SCHELE et A. KHOREVA, « A U-Net based discriminator for generative adversarial networks », *Computer Vision and Pattern Recognition*, 2020 (cf. p. 65).

[Schuhmann, 2021] C. SCHUHMANN, R. VENCU, R. BEAUMONT, R. KACZMARCZYK, C. MULLIS, A. KATTA, T. COOMBES, J. JITSEV et A. KOMATSUZAKI, « Laion-400m : Open dataset of clip-filtered 400 million image-text pairs », *arXiv preprint arXiv:2111.02114*, 2021 (cf. p. 71, 87).

[Shen, 2021] Z. SHEN, M. ZHANG, H. ZHAO, S. YI et H. LI, « Efficient attention : Attention with linear complexities », *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, 2021 (cf. p. 19, 27).

[Shin, 2016] H.-C. SHIN, H. R. ROTH, M. GAO, L. LU, Z. XU, I. NOGUES, J. YAO, D. MOLLURA et R. M. SUMMERS, « Deep convolutional neural networks for computer-aided detection : CNN architectures, dataset characteristics and transfer learning », *IEEE transactions on medical imaging*, t. 35, n° 5, p. 1285-1298, 2016 (cf. p. 1, 94).

[Sicuranza, 1990] G. SICURANZA, G. ROMPONI et S. MARSI, « Artificial neural network for image compression », *Electronics letters*, t. 26, n° 7, p. 477-479, 1990 (cf. p. 56).

[Silberman, 2012] N. SILBERMAN, D. Hoiem, P. KOHLI et R. FERGUS, « Indoor segmentation and support inference from rgbd images », *European Conference on Computer Vision*, 2012 (cf. p. 76, 83).
[Simonyan, 2014] K. SIMONYAN et A. ZISSERMAN, « Very deep convolutional networks for large-scale image recognition », *arXiv preprint arXiv:1409.1556*, 2014 (cf. p. 7).

[Singer, 2022] U. SINGER, A. POLYAK, T. HAYES, X. YIN, J. AN, S. ZHANG, Q. HU, H. YANG, O. ASHUAL, O. GAFNI et al., « Make-a-video: Text-to-video generation without text-video data », *arXiv preprint arXiv:2209.14792*, 2022 (cf. p. 1, 92, 94).

[Singh, 2023] M. SINGH, Q. DUVAL, K. V. ALWALA, H. FAN, V. AGGARWAL, A. ADCOCK, A. JOULIN, P. DOLLÁR, C. FEICHTENHOFER, R. GIRSHICK et al., « The effectiveness of MAE pre-pretraining for billion-scale pretraining », *arXiv preprint arXiv:2303.13496*, 2023 (cf. p. 93).

[Socher, 2014] R. SOCHER, A. KARPATHY, Q. V. LE, C. D. MANNING et A. Y. NG, « Grounded compositional semantics for finding and describing images with sentences », *Transactions of the Association for Computational Linguistics*, 2014 (cf. p. 72).

[Sonehara, 1989] N. SONEHARA, « Image data compression using a neural network model », *Proc. Int. Joint Conf. on Neural Networks*, 1989, p. II-35 (cf. p. 56).

[Song, 2015] S. SONG, S. P. LICHTENBERG et J. XIAO, « Sun rgb-d: A rgb-d scene understanding benchmark suite », *Computer Vision and Pattern Recognition*, 2015 (cf. p. 76, 83).

[Srinivas, 2021] A. SRINIVAS, T.-Y. LIN, N. PARMAR, J. SHLENS, P. ABBEEL et A. VASWANI, « Bottleneck transformers for visual recognition », *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 2021, p. 16519-16529 (cf. p. 11).

[Strüpler, 2022] Y. STRÜPLER, J. POSTELS, R. YANG, L. van GOOL et F. TOMBARI, « Implicit Neural Representations for Image Compression », *European Conference on Computer Vision*, 2022 (cf. p. 56).

[Sukhbaatar, 2019] S. SUKHBAATAR, E. GRAVE, P. BOJANOWSKI et A. JOULIN, « Adaptive attention span in transformers », *arXiv preprint arXiv:1905.07799*, 2019 (cf. p. 19).

[Sullivan, 2012] G. J. SULLIVAN, J.-R. OHM, W.-J. HAN et T. WIEGAND, « Overview of the high efficiency video coding (HEVC) standard », *IEEE Transactions on circuits and systems for video technology*, t. 22, n° 12, p. 1649-1668, 2012 (cf. p. 54).

[Szegedy, 2015] C. SZEGEDY, W. LIU, Y. JIA, P. SERMANET, S. REED, D. ANGUELOV, D. ERHAN, V. VANHOUCKE et A. RABINOVICH, « Going deeper with convolutions », *Computer Vision and Pattern Recognition*, 2015 (cf. p. 1, 7, 94).

[Tan, 2019] M. TAN et Q. LE, « Efficientnet: Rethinking model scaling for convolutional neural networks », *International Conference on Machine Learning*, PMLR, 2019 (cf. p. 8, 17, 29).

[Taubman, 2000] D. TAUBMAN, « High performance scalable image compression with EBCOT », t. 9, n° 7, p. 1158-1170, 2000 (cf. p. 54, 59).

[Taubman, 2012] D. TAUBMAN et M. MARCELLIN, « JPEG2000 Image Compression Fundamentals, Standards and Practice : Image Compression Fundamentals, Standards and Practice », *Springer Science and Business Media*, 2012 (cf. p. 61).

[Tay, 2020] Y. TAY, M. DEHGHANI, D. BAHRI et D. METZLER, « Efficient transformers : A survey. (2020) », *arXiv preprint cs.LG/2009.06732*, 2020 (cf. p. 10).

[Theis, 2017] L. THEIS, W. SHI, A. CUNNINGHAM et F. HUSZÁR, « Lossy image compression with compressive autoencoders », *International Conference on Learning Representations*, 2017 (cf. p. 56).

[Thomee, 2016] B. THOMEE, D. A. SHAMMA, G. FRIEDLAND, B. ELIZALDE, K. NI, D. POLAND, D. BORTH et L.-J. LI, « YFCC100M : The new data in multimedia research », *Communications of the ACM*, t. 59, n° 2, p. 64-73, 2016.

[Tian, 2020] C. TIAN, L. FEI, W. ZHENG, Y. XU, W. ZUO et C.-W. LIN, « Deep learning on image denoising : An overview », t. 131, p. 251-275, 2020 (cf. p. 54).

[Tian, 2019] Y. TIAN, D. KRISHNAN et P. ISOLA, « Contrastive Multiview Coding », *arXiv preprint arXiv:1906.05849*, 2019 (cf. p. 71-73, 86).

[Toderici, 2020] G. TODERICI, W. SHI, R. TIMOFTE, J. B. LUCAS THEIS, E. AGUSTSSON, N. JOHNSTON et F. MENTZER, *Workshop and Challenge on Learned Image Compression (CLIC2020)*, CVPR, 2020 (cf. p. 64).

[Tolias, 2016a] G. TOLIAS, Y. AVRITHIS et H. JÉGOU, « Image search with selective match kernels : aggregation across single and multiple images », *International journal of Computer Vision*, t. 116, n° 3, 2016 (cf. p. 101).

[Tolias, 2020] G. TOLIAS, T. JENICEK et O. CHUM, « Learning and aggregating deep local descriptors for instance-level recognition », *European Conference on Computer Vision*, 2020 (cf. p. 101-103).
[Tolias, 2016b] G. TOLIAS, R. SICRE et H. JÉGOU, « Particular Object Retrieval With Integral Max-Pooling of CNN Activations », *International Conference on Learning Representations*, 2016 (cf. p. 100).

[Tolstikhin, 2021] I. TOLSTIKHIN, N. HOULSBY, A. KOLESNIKOV, L. BEYER, X. ZHAI, T. UNTERTHINER, J. YUNG, A. STEINER, D. KEYSERS, J. USZKOREIT, M. LUCIC et A. DOSOVITSKIY, « MLP-Mixer : An all-MLP Architecture for Vision », *arXiv preprint arXiv:2105.01601*, 2021 (cf. p. 20).

[Tong, 2022] Z. TONG, Y. SONG, J. WANG et L. WANG, « VideoMAE : Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training », *Advances in Neural Information Processing Systems*, 2022 (cf. p. 5, 98).

[Touvron, 2019] H. TOUVRON, A. VEDALDI, M. DOUZE et H. JÉGOU, « Fixing the train-test resolution discrepancy », *Advances in Neural Information Processing Systems*, 2019 (cf. p. 22).

[Touvron, 2021a] H. TOUVRON, P. BOJANOWSKI, M. CARON, M. CORD, A. EL-NOUBY, E. GRAVE, A. JOULIN, G. SYNNAEVE, J. VERBEEK et H. JÉGOU, « ResMLP : Feedforward networks for image classification with data-efficient training », *arXiv preprint arXiv:2105.03404*, 2021 (cf. p. 20).

[Touvron, 2020] H. TOUVRON, M. CORD, M. DOUZE, F. MASSA, A. SABLAYROLLES et H. JÉGOU, « Training data-efficient image transformers and distillation through attention », *arXiv preprint arXiv:2012.12877*, 2020 (cf. p. 5, 11, 17, 18, 24-29, 40, 46, 48, 98).

[Touvron, 2021b] H. TOUVRON, M. CORD, M. DOUZE, F. MASSA, A. SABLAYROLLES et H. JÉGOU, « Training data-efficient image transformers & distillation through attention », *International Conference on Machine Learning*, 2021 (cf. p. 88).

[Touvron, 2022a] H. TOUVRON, M. CORD et H. JÉGOU, « Deit iii : Revenge of the vit », *Computer Vision-ECCV 2022 : 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV*, Springer, 2022, p. 516-533 (cf. p. 11).

[Touvron, 2022b] H. TOUVRON, M. CORD, A. EL-NOUBY, J. VERBEEK et H. JÉGOU, « Three things everyone should know about vision transformers », *Computer Vision-ECCV 2022 : 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV*, Springer, 2022, p. 497-515 (cf. p. 11).

[Touvron, 2021c] H. TOUVRON, M. CORD, A. SABLAYROLLES, G. SYNNAEVE et H. JÉGOU, « Going deeper with Image Transformers », *arXiv preprint arXiv:2103.17239*, 2021 (cf. p. 11, 17-19, 24, 25, 28, 29).

[Van Horn, 2018] G. VAN HORN, O. MAC AODHA, Y. SONG, Y. CUI, C. SUN, A. SHEPARD, H. ADAM, P. PERONA et S. BELONGIE, « The inaturalist species classification and detection dataset », *Computer Vision and Pattern Recognition*, 2018 (cf. p. 37, 45, 46).

[Vaswani, 2017a] A. VASWANI, N. SHAZEER, N. PARMAR, J. USZKOREIT, L. JONES, A. GOMEZ, L. KAISER et I. POLOSUKHIN, « Attention Is All You Need », *Advances in Neural Information Processing Systems*, 2017 (cf. p. 56, 63).

[Vaswani, 2017b] A. VASWANI, N. SHAZEER, N. PARMAR, J. USZKOREIT, L. JONES, A. N. GOMEZ, Ł. KAISER et I. POLOSUKHIN, « Attention is all you need », *Advances in Neural Information Processing Systems*, 2017 (cf. p. 1-3, 8, 9, 17, 20, 25, 35, 40, 52, 75, 94, 96, 97).

[Vincent, 2008] P. VINCENT, H. LAROCHELLE, Y. BENGIO et P.-A. MANZAGOL, « Extracting and composing robust features with denoising autoencoders », *Proceedings of the 25th international conference on Machine learning*, 2008, p. 1096-1103 (cf. p. 12, 38, 39).

[Vincent, 2010] P. VINCENT, H. LAROCHELLE, I. LAJOIE, Y. BENGIO, P.-A. MANZAGOL et L. BOTTOU, « Stacked denoising autoencoders : Learning useful representations in a deep network with a local denoising criterion. », *Journal of machine learning research*, t. 11, n° 12, 2010 (cf. p. 39).

[Wang, 2020a] G. WANG, J. C. YE et B. DE MAN, « Deep learning for tomographic image reconstruction », *Nature Machine Intelligence*, t. 2, n° 12, p. 737-748, 2020 (cf. p. 54).

[Wang, 2020b] H. WANG, Y. ZHU, B. GREEN, H. ADAM, A. YUILLE et L.-C. CHEN, « Axial-deeplab : Stand-alone axial-attention for panoptic segmentation », *Computer Vision-ECCV 2020 : 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV*, Springer, 2020, p. 108-126 (cf. p. 11).

[Wang, 2022] R. WANG, D. CHEN, Z. WU, Y. CHEN, X. DAI, M. LIU, Y.-G. JIANG, L. ZHOU et L. YUAN, « BEVT : Bert pretraining of video transformers », *Computer Vision and Pattern Recognition*, 2022 (cf. p. 72).

[Wang, 2020c] S. WANG, B. LI, M. KHABSA, H. FANG et H. MA, « Linformer : Self-attention with linear complexity », *arXiv preprint arXiv:2006.04768*, 2020 (cf. p. 10, 19, 27).

[Wang, 2021a] W. WANG, E. XIE, X. LI, D.-P. FAN, K. SONG, D. LIANG, T. LU, P. LUO et L. SHAO, « Pyramid vision transformer : A versatile backbone for dense prediction without convolutions », *arXiv preprint arXiv:2102.12122*, 2021 (cf. p. 11, 17, 19, 27, 33, 34).
[Wang, 2018] X. WANG, R. GIRSHICK, A. GUPTA et K. HE, « Non-local neural networks », *Computer Vision and Pattern Recognition*, 2018 (cf. p. 1, 2, 10, 94, 95).

[Wang, 2015] X. WANG et A. GUPTA, « Unsupervised learning of visual representations using videos », *Proceedings of the IEEE international conference on computer vision*, 2015, p. 2794-2802 (cf. p. 13).

[Wang, 2021b] Z. WANG, J. CHEN et S. C. H. HOI, « Deep Learning for Image Super-Resolution : A Survey », t. 43, n° 10, p. 3365-3387, 2021 (cf. p. 54).

[Wang, 2004] Z. WANG, A. C. BOVIK, H. R. SHEIKH et E. P. SIMONCELLI, « Image quality assessment : from error visibility to structural similarity », t. 13, n° 4, p. 600-612, 2004 (cf. p. 64).

[Wang, 2003] Z. WANG, E. P. SIMONCELLI et A. C. BOVIK, « Multiscale structural similarity for image quality assessment », *Asilomar Conference on Signals, Systems & Computers*, 2003 (cf. p. 61, 64).

[Wei, 2022] Y. WEI, H. HU, Z. XIE, Z. ZHANG, Y. CAO, J. BAO, D. CHEN et B. GUO, « Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation », *arXiv preprint arXiv:2205.14141*, 2022 (cf. p. 73).

[Wightman, 2019] R. WIGHTMAN, *PyTorch Image Models*, https://github.com/rweightman/pytorch-image-models, 2019 (cf. p. 25, 28).

[Witten, 1987] I. H. WITTEN, R. M. NEAL et J. G. CLEARY, « Arithmetic coding for data compression », *Communications of the ACM*, t. 30, n° 6, p. 520-540, 1987 (cf. p. 54).

[Wu, 2018a] Y. WU et K. HE, « Group normalization », *European Conference on Computer Vision*, 2018 (cf. p. 23).

[Wu, 2018b] Z. WU, Y. XIONG, S. X. YU et D. LIN, « Unsupervised feature learning via non-parametric instance discrimination », *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2018, p. 3733-3742 (cf. p. 40, 74).

[Xiao, 2018] T. XIAO, Y. LIU, B. ZHOU, Y. JIANG et J. SUN, « Unified perceptual parsing for scene understanding », *European Conference on Computer Vision*, 2018 (cf. p. 34, 47, 48).

[Xie, 2016] J. XIE, R. GIRSHICK et A. FARHADI, « Unsupervised deep embedding for clustering analysis », *International conference on machine learning*, PMLR, 2016, p. 478-487 (cf. p. 40).

[Xie, 2017] S. XIE, R. GIRSHICK, P. DOLLÁR, Z. TU et K. HE, « Aggregated residual transformations for deep neural networks », *Computer Vision and Pattern Recognition*, 2017 (cf. p. 33, 34).

[Xie, 2021] Z. XIE, Y. LIN, Z. YAO, Z. ZHANG, Q. DAI, Y. CAO et H. HU, « Self-Supervised Learning with Swin Transformers », *arXiv preprint arXiv:2105.04553*, 2021 (cf. p. 31).

[Xiong, 2021a] Y. XIONG, Z. ZENG, R. CHAKRABORTY, M. TAN, G. FUNG, Y. LI et V. SINGH, « Nyströmformer : A Nyström-Based Algorithm for Approximating Self-Attention », *arXiv preprint arXiv:2102.03902*, 2021 (cf. p. 19).

[Xiong, 2021b] Y. XIONG, Z. ZENG, R. CHAKRABORTY, M. TAN, G. FUNG, Y. LI et V. SINGH, « Nyströmformer : A nyström-based algorithm for approximating self-attention », *Proceedings of the AAAI Conference on Artificial Intelligence*, 2021 (cf. p. 10).

[Xu, 2022] H. XU, J. LI, A. BAEVSKI, M. AULI, W. GALUBA, F. METZE et C. FEICHTENHOFER, « Masked autoencoders that listen », *Advances in Neural Information Processing Systems*, 2022 (cf. p. 5, 79, 98).

[Xu, 2016] J. XU, T. MEI, T. YAO et Y. RUI, « Msr-vtt : A large video description dataset for bridging video and language », *Computer Vision and Pattern Recognition*, 2016 (cf. p. 77).

[Xue, 2022] H. XUE, Y. SUN, B. LIU, J. FU, R. SONG, H. LI et J. LUO, « CLIP-ViP : Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment », *arXiv preprint arXiv:2209.06430*, 2022 (cf. p. 72).

[Yang, 2016] J. YANG, D. PARIKH et D. BATRA, « Joint unsupervised learning of deep representations and image clusters », *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2016 (cf. p. 40).

[Yang, 2020] Y. YANG, R. BAMLER et S. MANDT, « Improving inference for neural image compression », *Advances in Neural Information Processing Systems*, 2020 (cf. p. 56).

[Yosinski, 2014] J. YOSINSKI, J. CLUNE, Y. BENGIO et H. LIPSON, « How transferable are features in deep neural networks? », *arXiv preprint arXiv:1411.1792*, 2014 (cf. p. 37).

[Youwang, 2022] K. YOUWANG, K. JI-YEON et T.-H. OH, « CLIP-Actor : Text-Driven Recommendation and Stylization for Animating Human Meshes », *ECCV*, 2022 (cf. p. 73).
[Yu, 2021] J. YU, X. LI, J. Y. KOH, H. ZHANG, R. PANG, J. QIN, A. KU, Y. XU et Y. W. JASON BALDRIDGE, « Vector-quantized Image Modeling with Improved VQGAN », *International Conference on Learning Representations*, 2021 (cf. p. 54, 58, 60, 63).

[Yu, 2022] J. YU, Z. WANG, V. VASUDEVAN, L. YEUNG, M. SEYEDHOSSEINI et Y. WU, « Coca : Contrastive captioners are image-text foundation models », *arXiv preprint arXiv:2205.01917*, 2022 (cf. p. 15, 71, 72).

[Yuan, 2021a] K. YUAN, S. GUO, Z. LIU, A. ZHOU, F. YU et W. WU, « Incorporating Convolution Designs into Visual Transformers », *arXiv preprint arXiv:2103.11816*, 2021 (cf. p. 19).

[Yuan, 2021b] L. YUAN, Y. CHEN, T. WANG, W. YU, Y. SHI, Z. JIANG, F. E. TAY, J. FENG et S. YAN, « Tokens-to-token ViT : Training vision transformers from scratch on ImageNet », *arXiv preprint arXiv:2101.11986*, 2021 (cf. p. 19).

[Yuan, 2021c] L. YUAN, D. CHEN, Y.-L. CHEN, N. CODELLA, X. DAI, J. GAO, H. HU, X. HUANG, B. LI, C. LI et al., « Florence : A new foundation model for computer vision », *arXiv preprint arXiv:2111.11432*, 2021 (cf. p. 71, 72).

[Yun, 2019] S. YUN, M. JEONG, R. KIM, J. KANG et H. J. KIM, « Graph transformer networks », *Advances in neural information processing systems*, t. 32, 2019 (cf. p. 5, 98).

[Zaheer, 2020] M. ZAHEER, G. GURUGANESH, A. DUBEY, J. AINSLIE, C. ALBERTI, S. ONTANON, P. PHAM, A. RAVULA, Q. WANG, L. YANG et al., « Big bird : Transformers for longer sequences », *arXiv preprint arXiv:2007.14062*, 2020 (cf. p. 9, 19).

[Zbontar, 2021] J. ZBONTAR, L. JING, I. MISRA, Y. LECUN et S. DENY, « Barlow twins : Self-supervised learning via redundancy reduction », *arXiv preprint arXiv:2103.03230*, 2021 (cf. p. 14, 40).

[Zhai, 2022] X. ZHAI, X. WANG, B. MUSTAFA, A. STEINER, D. KEYSERS, A. KOLESNIKOV et L. BEYER, « Lit : Zero-shot transfer with locked-image text tuning », *Computer Vision and Pattern Recognition*, 2022 (cf. p. 5, 15, 72, 98).

[Zhang, 2021] P. ZHANG, X. DAI, J. YANG, B. XIAO, L. YUAN, L. ZHANG et J. GAO, « Multi-Scale Vision Longformer : A New Vision Transformer for High-Resolution Image Encoding », *arXiv preprint arXiv:2103.15358*, 2021 (cf. p. 17, 20, 33).

[Zhang, 2018] R. ZHANG, P. ISOLA, A. EFROS, E. SHECHTMAN et O. WANG, « The Unreasonable Effectiveness of Deep Features as a Perceptual Metric », *Computer Vision and Pattern Recognition*, 2018 (cf. p. 62).

[Zhang, 2022] R. ZHANG, Z. GUO, W. ZHANG, K. LI, X. MIAO, B. CUI, Y. QIAO, P. GAO et H. LI, « Pointclip : Point cloud understanding by clip », *Computer Vision and Pattern Recognition*, 2022 (cf. p. 73, 74).

[Zhang, 2016] R. ZHANG, P. ISOLA et A. A. EFROS, « Colorful image colorization », *European conference on computer vision*, Springer, 2016, p. 649-666 (cf. p. 39).

[Zhao, 2020] H. ZHAO, J. JIA et V. KOLTUN, « Exploring Self-Attention for Image Recognition », *Computer Vision and Pattern Recognition*, 2020 (cf. p. 20).

[Zheng, 2020] S. ZHENG, J. LU, H. ZHAO, X. ZHU, Z. LUO, Y. WANG, Y. FU, J. FENG, T. XIANG, P. H. TORR et al., « Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers », *arXiv preprint arXiv:2012.15840*, 2020 (cf. p. 17, 20, 34).

[Zhong, 2020] Z. ZHONG, L. ZHENG, G. KANG, S. LI et Y. YANG, « Random erasing data augmentation », *Conference on Artificial Intelligence*, 2020 (cf. p. 86, 87).

[Zhou, 2014] B. ZHOU, A. LAPEDRIZA, J. XIAO, A. TORRALBA et A. OLIVA, « Learning Deep Features for Scene Recognition using Places Database », *Advances in Neural Information Processing Systems*, 2014 (cf. p. 77).

[Zhou, 2017] B. ZHOU, H. ZHAO, X. PUIG, S. FIDLER, A. BARRIUSO et A. TORRALBA, « Scene parsing through ade20k dataset », *Computer Vision and Pattern Recognition*, 2017 (cf. p. 34, 45, 46).

[Zhou, 2022] X. ZHOU, R. GIRDHAR, A. JOULIN, P. KRÄHENBÜHL et I. MISRA, « Detecting Twenty-thousand Classes using Image-level Supervision », *European Conference on Computer Vision*, 2022 (cf. p. 73, 81, 82).

[Zhu, 2021] Y. ZHU, Y. YANG et T. COHEN, « Transformer-based Transform Coding », *International Conference on Learning Representations*, 2021 (cf. p. 59).
<figure><img src="image_37.png" /></figure>
# RÉSUMÉ

Les transformateurs ont révolutionné l'apprentissage de la représentation dans de nombreuses modalités, obtenant des résultats de pointe dans le traitement du langage naturel, la vision par ordinateur, la parole et bien d'autres domaines. Cette thèse explore le potentiel des modèles de transformateurs pour la vision par ordinateur. Nous proposons des innovations architecturales pour surmonter certaines de leurs limites. Nous développons des méthodes de pré-entraînement auto-supervisé efficaces en termes d'échantillons, et considérons l'utilisation de ces transformateur dans un contexte d'apprentissage multimodal. Dans un premier temps, nous proposons l'attention à covariance croisée pour réduire la complexité quadratique de l'attention d'origine et obtenir des performances similaires avec une empreinte mémoire et un coût de calcul moindres, ce qui permet d'appliquer les transformateurs de vision à des images à plus haute résolution. Nous étudions ensuite le pré-entraînement auto-supervisé pour les transformateurs de vision. Nous proposons SplitMask, une méthode de débruitage automatique basée sur la modélisation d'images masquées. Contrairement aux méthodes de plongements conjointes, SplitMask ne nécessite pas d'ensembles de données de pré-entraînement à grande échelle et peut être appliqué à diverses données visuelles. SplitMask est aussi performant que les méthodes de plongements conjoints lorsqu'il est entraîné sur des ensembles de données deux fois plus petits, ce qui met en évidence l'amélioration de l'efficacité d'apprentissage avec peu de données. En outre, nous appliquons la modélisation d'images masquée à la compression d'images neuronales sous la forme d'un modèle entropique amélioré. Cela permet d'obtenir de bonnes performances en matière de débit-distorsion dans les régimes où la compression d'image est extrême, tels la taille d'un SMS ou d'un tweet. Enfin, nous proposons ImageBind, une méthode d'apprentissage d'un espace de plongement partagé entre six modalités. En résumé, cette thèse démontre le potentiel des transformateurs pour la vision par ordinateur grâce à des innovations architecturales, de nouveaux objectifs auto-supervisés et un transfert de connaissances multimodal. Les méthodes proposées dans cette thèse repoussent les limites des transformateurs en vision en améliorant leur passage à l'échelle et leur généralité, en permettant un apprentissage de la représentation plus efficace en termes d'échantillons, et en facilitant le transfert entre les modalités.

# MOTS CLÉS

transformateurs de vision, apprentissage auto-supervisé, apprentissage faiblement supervisé, apprentissage multimodal, compression d'image.

# ABSTRACT

Transformers have revolutionized representation learning across modalities, achieving state-of-the-art results in natural language processing, computer vision, speech, and beyond. This thesis explores the potential of Transformer models for computer vision. We propose architectural innovations to overcome their limitations, developing sample-efficient self-supervised pre-training methods, and advancing multimodal learning with Transformers. First, we propose Cross-Covariance Attention to reduce the quadratic complexity of self-attention achieving similar performance as vision transformers with lower memory footprint and computational cost, enabling the application of vision transformers to higher-resolution images. We then investigate self-supervised pre-training for vision transformers. We propose SplitMask, a denoising autoencoding method based on masked image modeling. Unlike joint embedding methods, SplitMask does not require large-scale pre-training datasets and can be applied to diverse visual data. SplitMask matches the performance of joint embedding methods when pre-trained on datasets two orders of magnitude smaller, highlighting its improved sample efficiency. Moreover, we apply masked image modeling to neural image compression in the form of an improved entropy model yielding a strong rate-distortion performance and enabling the compression of images to the size of a short SMS or tweet. Finally, we propose ImageBind, a method for learning a shared embedding space across six modalities. Image-Bind leverages the abundance of images and text on the web to enable transfer to modalities with scarce annotations like depth, thermal, audio, and IMU. In summary, this thesis demonstrates the potential of Transformers for computer vision through architectural innovations, new self-supervised objectives, and multimodal knowledge transfer. The methods proposed in this thesis push the boundaries of transformers in vision by enhancing their scalability and generality, enabling more sample-efficient representation learning, and facilitating transfer across modalities.

# KEYWORDS

vision transformers, self-supervised learning, weakly supervised learning, multimodal learning, image compression