# Fine-Tuned Visual Transformer Masked Autoencoder Applied for Anomaly Detection in Satellite Images
## 1 Introduction  
When one considers the applications of the Transformer Model [1], computer image analysis is probably not the first thing that comes to mind. Transformer Networks and Transformer-based models increased in popularity after the Large Language Model solution called GPT-1 was released. In addition, text-to-image solutions (e.g., DALL-E) were built on OpenAI's model [2, 3, 4].  
The anomaly detection process, also referred to as outlier or novelty detection, has been characterized as a recent trend in machine learning tasks by many research groups [5, 6, 7, 8]. The concept involves the process of looking for samples of data that are located outside the assumed 'normal' class. The size of the area created in this process can be defined using various metrics and techniques. Novelty detection can be considered both as a part of the data preparation step within the machine learning workflow and a separate machine learning task in addition to regression, classification, and reinforcement learning [9]. There are many different anomaly detection techniques, such as rule-based methods, probability-based methods [10, 11], distance-based methods [12, 13, 14, 15, 16, 17], and neural-network-based approaches [5, 18, 19, 20, 21, 22]. The final group seems to stand out from the aforementionedones, as the range of problems where outlier detection is performed with the help of artificial neural networks has been increasing in recent times. In this paper, only computer image datasets and related applications are relevant. In areas where novelty detection has emerged as an important task, such as surface inspection [23] and medical image analysis [24; 25; 26; 27], applications connected to cybersecurity [8; 28] can be recognized.  
Regarding the motivations for the idea of this paper, to our best knowledge, there are only a few papers that have addressed solutions to the problem of anomaly detection for satellite image datasets. Possible applications of such a solution may include a wide range of fields such as military applications, environmental engineering, tracking and detection, and biology and environmental protection. Satellite image analysis can be easily transformed into aerial and unmanned aerial vehicle image analysis depending on current needs and possibilities in terms of the available data sources. A few factors make this task complex. First, the model is primarily based on image spatial resolution. In georeferencing, this can be explained as the smallest possible unit of a given image that is capable of being distinguished or the smallest possible angular or linear distance to identify adjacent objects. The resolution of the image and the level of detail in the images may be general problems for all computer vision tasks. In terms of anomaly detection for satellite vision, another issue exists, i.e., the homogeneity of terrain. Characterizing homogeneity as a high level of similarity between consecutive snapshots of a given piece of land, we can assume that the set of possible normal images of a given environment may differ significantly and would be based on the subjective knowledge and purpose of the analysis. For the purpose of this research, normal samples comprising the natural environment without buildings in the surroundings or man-made architecture without signs of natural disasters or war-related changes were considered normal. This problem can later be addressed further when dealing with a specific task type. Whether we want to inspect natural habitats, urban areas, or security-sensitive areas, datasets can be modified to reflect actual needs. In addition, clouds and weather conditions can obscure the ground, and the detection of anomalies in certain areas could be a challenging task. The assumption for this paper is to construct and test the general requirements and broad sense of concepts for anomalies and normal samples.