# Evaluation of StyleGAN-CLIP Models in Text-to-Image Generation of Faces
## 5 Results and Discussion  
In this section, we summarise and discuss the results. We compute inter-rater agreement scores to inform our confidence in human ratings, and we study the correlation between human ratings and automatic metrics on the various models. In addition, we compare editing and non-editing models in terms of the final image, and we compare models on the basis of computational time. Inter-rater Agreement: Table 4 gives the per-question, per-model and overall mean score, standard deviation and intraclass correlation (ICC) coefficients. The overall ICC for Q1 is 0.78, indicating good inter-rater agreement; the ICC for Q2 is 0.63, i.e., moderate inter-rater agreement; whilst the ICC for Q3 is 0.55, moderate (fair) inter-rater agreement. Overall, the inter-rater agreement interpretation is moderate to good. The raters agreed mostly on the first question, i.e., the extent to which the image matches the description, whilst the second and third questions are probably more subjective. \begin{table}
\begin{tabular}{l l c c c c c c} \hline \hline  & **Metric** & **StyleGAN** & **StyleGAN2** & **StyleGAN2** & **StyleGAN2** & **StyleGAN3** & **Overall** \\ \hline \multirow{4}{*}{Q1} & Mean & 74.4 & 81.2 & 4.0 & 45.3 & 71.4 & 55.3 \\  & Std Dev & 18.8 & 16.5 & 5.0 & 20.8 & 18.9 & 33.0 \\  & \(ICC_{\textit{Cor}}\) & & & & & & 0.78 \\ \hline \multirow{4}{*}{Q2} & Mean & 15.7 & 67.9 & 87.7 & 71.2 & 63.4 & 61.2 \\  & Std Dev & 15.9 & 24.4 & 16.3 & 21.7 & 23.9 & 31.8 \\  & \(ICC_{\textit{Cor}}\) & & & & & & 0.63 \\ \hline \multirow{4}{*}{Q3} & Mean & 43.2 & 74.4 & 18.3 & 47.8 & 64.8 & 49.7 \\  & Std Dev & 22.2 & 18.6 & 15.7 & 21.5 & 19.2 & 27.4 \\ \cline{1-1}  & \(ICC_{\textit{Cor}}\) & & & & & 0.55 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Inter-rater agreement: Mean score, standard deviation and intra-class correlation (\(ICC_{\textit{Agreement}}\), \(ICC_{\textit{Consistency}}\)) for Q1: “To what extent does the image match the description”, Q2: “To what extent does the image look like a real photo” and Q3: “How satisfied are you with this image, considering both how much it matches the description and the quality of the image”. Correlation between human ratings and metrics: Figure 2 shows the linear relationships between the human ratings and automatic metrics (CLIP (r = 0.065, \(p\) = 0.005), FID (r = \(-\)0.005, \(p\) = 0.822) and CMMD (r = \(-\)0.264, \(p\) = 0.0)). All values indicate no or very weak linear relationships between the variables. R values tend to be positive for Q1-CLIP but negative for both Q2-CMMD and Q2-FID, as expected. In addition, CMMD is a better model of human ratings for Q2, when compared to FID. However, it is clear that automatic metrics require further study and development. Thus, we consider human ratings as more reliable. Inferring Q3 rating from Q1 and Q2 human ratings: Automatic metrics usually only check for Q1 or Q2, but we also want to know the answer to Q3. We were therefore interested in knowing whether the overarching 'Q3 score' can be predicted from the simpler scores (Q1 and Q2). To predict the 'Q3 score', we considered a weighted average of Q1 and Q2 scores (given by human raters), computed as \(\hat{Q3}=\lambda\times Q1+(1-\lambda)\times Q2\), and maximized the correlation coefficient (Q3 vs \(\hat{Q3}\)) via a line search that yielded \(\lambda=0.65\). Figure 3 depicts the correlation between 'Q3 score' and 'Q1 score' and 'Q2 score' separately, and the 'combination of Q1 and Q2 score'. The correlation coefficient for the latter is (r = 0.91, \(p<0.0001\)) and, therefore, we conclude that to predict the 'Q3 score', it is sufficient to take a weighted average, given good automatic metrics from Q1 and Q2. The weighting coefficient \(\lambda\) is not 0.5 (i.e., Q1 and Q2 have equal weighting), probably because Q1 correlates with Q3 more than Q2 correlates with Q3. This can be interpreted that humans value Q1 more than Q2 overall.