## 5 Impact Sound Generation for Object Materials
### Proposed GAN Architecture
This makes it easier to control because the loss function is only adversarial loss, unlike with AC-GAN. Ref. [34] is also aimed at super-resolution image generation by label classification, as in the AC-GAN GAN architecture, and when the conditional GAN model is applied, the accuracy of label classification is higher than that of the conventional concatenation-based conditional GAN, achieving higher quality, super-resolution generation. [PERSON] et al. came up with an architecture, based on Residual Block. Based on their research results, we found that it should be possible to introduce it into the architecture of this research and, thereby, to improve the accuracy for the task of generating the impact sound for each material label. By introducing Projection, we can generate spectrogram images of various classes and represent material classes continuously, so if the input is not one-hot, that is, the difference between classes, learning can also be expected in spectrogram image generation, in which the estimation result of the material class is directly input. Figure 11: Proposed architecture of GAN. (**a**) generator network, (**b**) discriminator network. For image generation, our method performed learning using the architecture based on Residual Block. Adversarial loss and gradient penalty [31] were used as the loss functions during learning based on DRAGAN [35]. Each weight is 1.0, for adversarial loss, and 0.5, for gradient penalty. We used Adam (\(\alpha=0.0002\), \(\beta 1=0.5\)) [36] as the weight optimization method. Before performing the spectrogram image generation experiments, it was necessary to create a dataset using the generative model. We first describe the restrictions on the dataset and the material label (class) we used, and then present the algorithm for spectrogram image generation. When learning, using a deep learning model, it is important to use a dataset suitable for the purpose. In addition, with learning aimed at generating different data, as in this study, it is important that the data distribution in the dataset, that is, the interclass and intraclass variance of the dataset, be suitable for the application. Therefore, we needed to determine the validity of the dataset to be used as training data. The proposed method utilized the Greatest Hits dataset to generate spectrogram images, based on material labels. In this dataset, as the material label is annotated with the sound of each action of the object in the video, we considered it suitable for the purpose of learning. As mentioned earlier, this dataset contains 17 classes of material data, and in theory, all of them can be trained. However, learning becomes difficult, depending on the distribution of data in all classes. Therefore, to investigate the variance in the data used for learning, we visualized the training data, based on TactGAN. Principal component analysis (PCA) was used as the method of dimensional compression of images for visualization. PCA is an effective tool for investigating the distribution of images and is mainly utilized for visualizing the similarity between multiple image data and clustering results. The method of converting audio data into images is described later. Figure 12 shows the results of visualizing 100 items of data, randomly acquired from each class in the Greatest Hits dataset. As we can see, learning is expected to be difficult here because there are no clear features in the distribution between the data in all 17 classes. This is why we decided to use six classes (cloth, glass, metal, paper, plastic, and wood) for the training of the material estimation model of the texture image. Figure 13 shows the results of the same analysis using the data of these six classes. Figure 12: Data distribution of 17 material classes. As we can see, the variance between classes here is easier to characterize, compared to in Figure 12. A few of the classes have similar variances, but to some extent, this is unavoidable, due to the characteristics of the impact sound data.