# Evaluation of StyleGAN-CLIP Models in Text-to-Image Generation of Faces
## 4 Evaluation Methodology
### Data Generation and Organisation  
In this study, eight StyleGAN-based models were evaluated in the text-to-image generation or image editing tasks. We generated a total of 1842 real human face images using 141 different textual descriptions to evaluate the different models. The descriptions were taken from the Face2 Text 2.0 dataset [48]. The descriptions consist of 69 single-sentence descriptions and 72 two-sentence descriptions. To ensure a wide variety of text lengths, the descriptions were sampled in a stratified way according to the number of tokens they were represented in (based on the CLIP tokeniser). The generated images were split into two groups: The first consists of 1410 face images synthesised from the five StyleGAN-based generation models: StyleGAN [18], StyleGAN2 [19], StyleGAN2-Adaptive [25], StyleGAN2-Distillation [26] and StyleGAN3 [20]. In this group, we used all 141 descriptions, and each description was used to generate two different images. The second group consists of 432 edited images, produced by three StyleGAN-based models for face image editing: StyleGAN2-editing [21], StyleGAN3-editing [27] and the StyleGAN2-CLIPInverter model [23]. The 72 two-sentence descriptions were used to generate the images in this group, such that the first sentence was used to create an image and the second sentence was used to edit the synthesised image. As in group one, each description was used to produce two different edited images. The generated images were organised in 47 batches, ready for evaluation by 47 unique human raters. Each batch consisted of 39 images, 30 of which are synthesised images, whilst the remaining 9 are edited images. A sample of generated images can be found in Table 1. To strike a balance between coverage and cost of evaluation, we opted for a larger number of batches, rather than a smaller repeated number and inserted an inter-annotator subset. The batches were organised using a Latin square design, such that each evaluator saw the same number of images generated by each model from unique descriptions. In addition, we added 13 control images to each batch, and these are the same across all batches. \begin{table}
\begin{tabular}{c c} \hline \hline  & **Description 1** & **Description 2** \\ \hline StyleGAN1 & \\ StyleGAN2 & \\ StyleGAN2 & \\ StyleGAN2 & \\ adaptive & \\ StyleGAN2 & \\ distilled & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Example images that were generated by different models using a description. Each model is used twice on the same description to generate two different images. The descriptions are: (1) “A young woman with a strong jaw and an otherwise round face. She has long, straight, brunette hair, full lips and brown eyes.” (2) “A tanned young boy with a long, slightly wavy bowl haircut. He has a wide nose, full cheeks and a toothy smile.” Edited models are used by first generating an image using the first sentence and then editing the image using the second sentence. Eight of the control images were used for quality control purposes. As explained further in the next section, the human evaluation consists of asking how realistic the image looks and how well it matches the description. Therefore, to test the quality of the evaluators' work, we included some vector graphic images of faces which are unambiguously unrealistic and some real photos of faces, which should be considered realistic, and these images are either paired with an unambiguously wrong description or an unambiguously correct description. We then checked that these are evaluated as expected and rejected the batch of evaluations if they failed the test. The fake images were all taken from the free stock photo website Pixabay ([[https://pixabay.com/](https://pixabay.com/)]([https://pixabay.com/](https://pixabay.com/))), which provides images with a permissive license, while the real face photos were taken from the FFHQ dataset [18]. This collection of 8 images with the descriptions used can be seen in Table 2.  
The remaining 5 control images form the inter-annotator subset.