## 5 Experiments
### Synthetic Data Generators Assessment  
Before starting the Algorithm selection evaluation, we evaluate the performance of the data generator networks. Figure 4(a),b show two common measures used to asses the quality of generated images. Figure 4(a) shows the Inception Score (IS) measure [53] applied to the pneumonia synthetic images generated by the used GAN. The IS metric is used to assess how sharp and distinct the generated images are. IS is calculated using the Inception v3 [54] pretrained image classification model applied to images generated by GAN. The score is maximized when two conditions are met. The first condition is high confidence in the label assigned to each image, indicating image sharpness. The second condition is the variety of the labels assigned to different images, indicating a high diversity of the generated images. Figure 4(b) shows the FID score. The FID score [52] is a measure of the distribution of the synthetic images when compared to the distribution of the real images. In particular, the covariance and the mean of the synthetic distribution of images are compared with the distribution parameters of the real images. Figure 4: The correlation between the five individual Algorithms trained for five epochs and evaluated on the \(\mathbb{D}_{T}\) dataset. Figure 5: Graphs of FID and IS metrics for GAN generated synthetic images based on the number of images used for training. Observe that both measures confirm that at least statistically the evaluated GAN converges properly or as expected. In other words, as more seed images are used to train the GAN, the generated samples more closely follow the original distribution of the real images, and the synthetic images become increasingly similar to the seed images. Note that we are interested not in generating synthetic samples highly similar to the distribution of real images, but rather we aim to capture the noise required for efficient generalization. While the FID score near 0 indicates the highest similarity to the original real-world images, it also implies that for in order to capture the noise beneficial for Algorithm selection, fewer images in the seed dataset \(\mathbb{D}_{E}\) are desired. Finally, we also evaluated the quality of synthetic images in a classification setting. For this purpose, we evaluated the accuracy of the five component Algorithms that were trained on images from \(\mathbb{D}_{l}\) for 5 epochs on the \(\mathbb{D}_{S}\) data generated by the GAN from various sizes of the \(\mathbb{D}_{E}\) dataset. Figure 6 shows the individual Algorithm accuracy as a function of the size of the \(\mathbb{D}_{E}\) dataset. Figure 6 shows that when the GAN is seeded with over 300 real images, the resulting synthetic dataset provides enough information to result in an accuracy over 80% for at least certain Algorithm. When seeded with 900 images, the accuracy of all Algorithms is larger than 80%. This means that the larger the seed dataset is, the greater the ability of the GAN to generate images similar to the seed dataset. This indicates that when the GAN is provided with too many images, its statistical output distribution converges to the original distribution of the seed dataset. While this result is expected [31], in this work we are looking for the complementarity of the Algorithm selection performance to the performance of the individual Algorithms. This means that we are looking for a complementary distribution of data samples that would allow us to predict the Algorithm accuracy \(\mathcal{A}\) on \(\mathbb{D}_{T}\) from \(\mathbb{D}_{E}\) using the smallest possible set of seed samples. Figure 7 shows images both real as well as synthetic ones. Figure 7a,b show examples of healthy and diagnosed images from the original dataset. Figure 7c,d show examples of synthetic images representing healthy individuals. Figure 7c show the generated images for the highest FID score, while the Figure 7d shows generated images for the best IS score.