# Syn2 Real Detection in the Sky: Generation and Adaptation of Synthetic Aerial Ship Images
## 3 Methods
### Image Generation with Models  
The DALL-E series [64, 65] from OpenAI, specifically the generative model called DALL-E by Midjourney Labs, and Stability AI's Stable Diffusion [66] are all highly regarded image generation models that are widely used in the field. For this paper, we selected Stable Diffusion as the generation tool due to its open-source nature. Stable Diffusion [66] is capable of generating images based on text and modifying existing images using input text. It consists of two main components: a text underwater and an image generator. The text underwater converts textual information into a numerical representation, allowing the model to understand the meaning of the text. It is implemented as a specially designed transformer language model, which is essentially a CLIP [67] model. When text is input into the text underwater, each word or token is associated with a vector feature, resulting in a feature list. This feature list serves as input information for the image generator. The image generator can be further divided into two parts: the image information creator and the image decoder. The image information creator operates in the latent space, which is crucial for Stable Diffusion. It consists of a UNet [68] neural network and a scheduling algorithm. The UNet network is responsible for transforming the arrays of latents. In Stable Diffusion, attention layers are added between the ResNet blocks of the UNet to process text tokens. The feature list is progressively denoised to transform it into an array of information, which is then passed to the image decoder. The image decoder generates an image based on the information provided by the image information creator. Figure 5 provides a simplified illustration of the image generation process using Stable Diffusion. Figure 4: The bounding box relative images sizes: (**a**) SASI-UNITY dataset, (**b**) SASI-ZERO dataset, and (**c**) SASI-RANDOM dataset. Figure 3: Location distribution of the instances: (**a**) SASI-UNITY dataset, (**b**) SASI-ZERO dataset, and (**c**) SASI-RANDOM dataset. There are two popular open-source versions of Stable Diffusion, namely version 1.5 and version 2.1. In practical application scenarios, version 2.1 is more industrialized and productive, while version 1.5 excels in terms of overall coherence and contextualization. Since they are trained on different bases, version 2.1 provides checkpoint files that enable the generation of 768-resolution images and improve performance on larger formats, which aligns better with the characteristics of remote sensing images. The demand for GPU memory is substantial when generating synthetic images with a resolution of \(768\times 768\). We set the model's batch size to 1, allowing the generation of one image per batch. The generation frequency needs to be adjusted to complete the generation of a large number of images. Stable Diffusion employs Classifier-Free Guidance (CFG) [69] to train the conditional diffusion model, and noise is introduced during the sampling stage. The formula used is as follows:  
\[n =\omega n_{c}+(1-\omega)n_{u} \tag{1}\] \[=\omega e_{\theta}(x_{t},t,c)+(1-\omega)e_{\theta}(x_{t},t),\]  
where \(n\) is noise, \(n_{c}\) is the conditional prediction of noise, \(n_{u}\) is the unconditional prediction of noise, \(e\) is the Gauss noise, \(x_{t}\) is the sample obtained from the real data sampled by time, \(t\) is the time step, \(c\) is the token embeddings, and \(\omega\) is the guidance scale. As \(\omega\) becomes bigger, the condition plays a bigger role, i.e., the generated image is more consistent with the input text. However, when the CFG [69] value is large enough, the image will distorted, so we consistently used a value of 9.0. According to the official example of Stable Diffusion, each run generates different depictions of the same input prompt statements. To enhance the diversity of the generated images and capture the complexity and variability of actual remote sensing images, we introduced a randomized prompt statement module.