# Evaluation of StyleGAN-CLIP Models in Text-to-Image Generation of Faces
## 3 Review of Text-to-Image Evaluation Metrics
### Automatic Evaluation Metrics  
In this subsection, we explore a number of automatic metrics to measure the text-image alignment.  
CLIP score: The CLIP model score [23; 34; 35] measures the compatibility of text-image pairs by computing the cosine similarity between the embedding of the generated images and the embedding of the text prompts. Higher CLIP scores signify higher semantic similarity between the image and the given description. CLIP scores were found to have a high correlation with human judgement [36].  
AugCLIP score: is a variant of the standard CLIP score and is more robust against adversarial attacks [37]. AugCLIP introduces random augmentations on the image, such as random colourisation, random translation, random resize and random cutout. Since the CLIP model is trained on various image formats, the augmentation strategy does not destroy the semantic features encoded by CLIP.  
R-Precision (RP): RP [38; 39; 40] is a popular metric that computes the top-R text-to-image retrieval accuracy. The goal of this metric is to retrieve the matching text from 100 candidate texts using the synthesised image as a query and the cosine similarity between the image encoding vectors and the text encoding vectors as a similarity score. An adapted version of the Deep Attentional Multimodal Similarity Model (DAMSM) [39] is used to compute the image-text similarity score for retrieval.  
CLIP-R-Precision: [PERSON] et al. [40] observed that models not specifically tuned on Deep Attentional Multimodal Similarity Models (DAMSMs) tend to perform poorly on the RP metric. To address this issue, the CLIP model [24] is used in place of the standard DAMSM to calculate the R-Precision scores. Compared to DAMSM, CLIP obtains higher image-text retrieval performance.  
TIFA (Text-to-Image Faithfulness evaluation with question Answering): TIFA [41] is a recent automatic evaluation metric that evaluates the correctness between the text and the image using a visual question answering (VQA) model. In addition to the VQA model, TIFA integrates two other models: a question-generating (QG) model and a question-answering(QA) model. Given a text input, the QG model generates several question-answer pairs, and the QA model filters them. Finally, the VQA model provides answers to the questions using the synthesised image and the answers are checked for correctness. To facilitate comparative studies, a TIFA benchmark is provided. This includes 4k text prompts in addition to 25k questions. TIFA is considered much more accurate than CLIP and has better correlation with human assessment. However, we were not able to use TIFA in our study since a suitable VQA model specifically for face descriptions was not readily available.  
FID (Frechet Inception Distance): FID [42] quantifies how similar the generated images are to real ones by comparing the distributions of feature representations, and as such, it is suitable to capture how realistic the generated image is. Both real and generated images are passed through a pretrained Inception-v3 [43] network, and the resulting feature vectors are assumed to follow multivariate Gaussians. FID computes the Frechet (or Wasserstein-2) distance between these two Gaussians. A lower score is therefore better.  
CMMD (CLIP Maximum Mean Discrepancy): CMMD [44] is similar in concept to FID; however, CMMD does not assume image feature vectors to be multivariate Gaussians and instead uses Maximum Mean Discrepancy (MMD) to measure the distance between CLIP embeddings instead of Inception-v3 feature vectors. The authors argue that CLIP embeddings should perform better since these are trained on 400 million image-text pairs instead of the 1M images used to train the Inception-v3 model. It is shown that CMMD correlates better with human evaluations when used to predict image realism.