## 5 Impact Sound Generation for Object Materials
### Sound Generation from Spectrogram Images  
As discussed in the previous section, it is necessary to convert the spectrogram image, generated by the generative model, into sound data, in order to present the impact sound. To create a dataset for learning the generative model, we converted the sound data of the impact sound into the generation of the spectrogram image. At that time, the time-series data was transformed into an amplitude spectrogram, obtained by STFT. In this amplitude spectrogram, the characteristic structure of sound data tended to appear, which was useful for processing and compositing in many situations. In this study, as well, we used the amplitude spectrogram as image data, to generate an image using a generation model.  
In order to present the impact sound to the user, based on the generated spectrogram image, it is necessary to convert the spectrogram image into sound data. As mentioned earlier, the spectrogram is composed of time, frequency, and amplitude elements for time-series data, but lacks phase information. Therefore, in order to reconstruct the sound data signal from the spectrogram, we restore the phase by using the Griffin-Lim algorithm [3].  
The conversion from sound data to spectrogram was performed as follows. First, the result \(X(m,\ k)\) of the STFT was depicted as a complex number. In addition, each \((m,\ k)\) was provided with an amplitude \(A(m,\ k)=|X(m,\ k)|\) and a phase \(\phi(m,\ k)=X(m,\ k)\). Amplitude \(A(m,\ k)\) was called an amplitude spectrogram, phase \(\phi(m,\ k)\) was called a phase spectrogram, and the power spectrogram, calculated by Equation (8), was used to show the intensity at each frequency component.  
\[|X(m,\ k)|^{2}=A^{2}(m,\ k) \tag{8}\]  
The spectrogram image used in the learning is this power spectrogram, which was equivalent to the calculation of the magnitude of real and complex vectors from the STFT results. That is, it had \(|X(m,\ k)|\) as the amplitude, but \(\phi(m,\ k)\), as the phase, was lost when converting sound data into a spectrogram. The Griffin-Lim algorithm restores the phase by STFT and inverse STFT.  
The amplitude spectrogram \(X(m,\ k)\) was obtained by performing STFT, using the window function \(w_{a}(t)\). In the Griffin-Lim algorithm, we wanted to satisfy \"the phase is consistent with the time signal\" from this amplitude spectrogram and proposed the following iterative method for doing so. First, an appropriate initial phase \(\phi(m,\ k)\) was selected for a given amplitude \(A(m,\ k)\). Then, after creating \(X(m,\ k)=A(m,\ k)\exp(j\phi(m,\ k))\) as a spectrogram, based on the initial phase, the following iterative processing was applied.  
\[x(t)\leftarrow\text{ISTFT}[X(m,\ k)] \tag{9}\]  
\[X(m,\ k)\leftarrow\text{STFT}[x(t)] \tag{10}\]  
\[X(m,\ k)\gets A(m,\ k)X(m,\ k)/|X(m,\ k)| \tag{11}\]  
\(X(m,\ k)\), created by giving the initial phase, as described above, is an expression that estimates the correct phase, based on the assumption that the phase is consistent with the time signal. This means that restoration is impossible, even if the inverse STFT and STFT are performed. Therefore, in this iterative method, the phase is obtained with the constraint that the correct STFT expression is obtained while fixing the amplitude to \(A(m,\ k)\) from Equation (9). In this way, the generated image is converted into sound data, and the result is presented to the user.