# Conditional Generative Adversarial Networks for Domain Transfer: A Survey
## 3 The Evaluation Method of CGAN
### Manual Evaluation  
Manual evaluation is a supplement to automatic evaluation, which is helpful for finding out some factors that are not considered enough in the process of designing automatic indexes. One of the simplest methods is to select some typical samples from the generated data for human analysis [23; 61; 64]. However, this simple analysis method is only representative of the authors' or recruited analysts' opinions, with a large degree of randomness.  
In order to make the evaluation results more objective, it is often necessary to invite several evaluators to evaluate independently. For example, in [64], several groups of images will be displayed, each of which contains a real image and a generated image, and the evaluator needs to choose the more likely real image based on his own judgment. There are several improved versions of this evaluation method. From the perspective of the evaluator, if the image involves professional content, the evaluator should be a domain expert [23]. From the perspective of the measurement scale, if we desire the evaluators to pay more attention to the overall image rather than local, the display time of the image can be limited (such as 1 second), and when the display time is depleted, the image immediately disappears. At this time, the evaluator needs to rely on the impression of the two images to choose the proper one [35]. From the point of view of evaluation results, if we want the evaluators to understand the attributes of real images to a deeper extent, there are mainly two schemes. In one way, we can demonstrate the answer to the evaluator at the beginning of the evaluation as feedback [35]. For another, we can display two real images and a generated image once at a time so that the evaluator will have more opportunities to perceive the attributes of the real image [65]. In addition, comparative experiments can also be conducted, in which the image generated by another algorithm replaces the real image, and the evaluator will choose the better image to attain the conclusion on which algorithm is better.  
Similar to automatic evaluation, in the task of text-to-image generation, there are corresponding manual evaluation methods in order to determine whether the generated image is conditional on the text. For example, in [60], a number of text descriptions are randomly selected for each category of data, and images are generated using different algorithms. The evaluator sorts these images from text-related to non-related, and then the average ranking of each algorithm is counted and compared.  
The manual evaluation effectively alleviates the problem that the evaluation index is out of line with the actual effect, and it is easier to grasp the overall effect of the generated data rather than the local performance. However, its disadvantages are also apparent, the main one of which is the consumption of more manpower. In addition, due to the influence of personal preferences, the evaluation results also have larger uncertainty.  
As a summary, we list the descriptions of various evaluation methods in Table 3. Because of the advantages and disadvantages of automatic evaluation and manual evaluation, they are often used simultaneously in many scenarios, playing a complementary role. When the result of the manual evaluation is significantly worse than that of automatic evaluation, it is necessary to consider whether the automatic evaluation metrics cannot reflect all aspects of the generated data. On the contrary, it is necessary to pay attention to whether the evaluators in the manual evaluation method are deceived by some appearance of the generated data, that is, fail to pay attention to some defects reflected by the automatic evaluation metrics.