# Fine-Tuned Visual Transformer Masked Autoencoder Applied for Anomaly Detection in Satellite Images
## 1 Introduction
### Related Work  
There are various approaches to detecting anomalies in terms of the methods used. A good review of the available studies regarding deep learning-based anomaly detection methods for reconstruction-based and prediction-based approaches in modeling complex data distributions is provided in [29]. Most reconstruction-based anomaly detection models have been built using techniques like Generative Adversarial Networks (GANs), autoencoders, and diffusion models. It should be noted that diffusion models have emerged as a new approach instead of traditional autoencoder-based sub-networks for reconstruction [30].  
As mentioned, one group used the Generative Adversarial Network approach to solve the problem without the need to label the data [31; 32]. In [33], a framework leveraging GAN inversion for high-quality feature reconstruction was proposed for anomaly detection. Another group employed the autoencoder concept (AE), as in, e.g., [34; 35; 36]. The autoencoder is a deep learning architecture trained with the objective of reproducing the input by the output layer. A typical AE comprises bottleneck layers that tend to reduce toward the center of the network. The encoder-decoder structure is obtained by visually splitting these layers into two halves. Each consecutive layer from the encoder has a lower dimensionality than the preceding layer. The goal is to extract the feature representation. Thus, each layer of the first half usually has its equivalent in the decoder. The goal of the decoder is to reproduce the input so that the output is as close as possible to the original input.  
As a subset of reconstruction-based methods, reconstruction combined with patch masking can be delineated. In this approach, a given part of the image is masked according to the defined masking ratio. The location of masked patches is generated randomly. With this applied image operation, an image is passed to the model, where non-masked patches are reconstructed in an autoencoder manner. The reconstructed patches are then integrated with the original patches. The position of the masked and non-masked patches remains invariant throughout the entire process. The loss function is computed as the pixel-wise difference between the original and reconstructed images. Typically, the autoencoder is later decomposed into encoder and decoder components, with the pre-learned encoder used for the image classification task [37].  
Anomaly detection within satellite images is quite a new field of research, as state-of-the-art analysis has shown. To our best knowledge, only a few papers have dealt with this topic. In [38], a statistical approach was applied to find anomalous samples. After analysing the vegetation index data for crop images, deviations from normal distributions could be detected. Regarding the application of neural networks, a framework was proposed in [39]. In this model, the artificial neural network model was applied to novelty detection. The detection process was based on well-designed features such as mean, standard deviation, entropy, white, and nonzero-pixel ratios that were calculated for each image, which allowed us to obtain input for a multi-layer perceptron model.