Conditioning Trick for Training Stable GANs

Abstract: In this paper we propose a conditioning trick, called difference departure from normality, applied on the generator network in response to instability issues during GAN training. We force the generator to get closer to the departure from normality function of real samples computed in the spectral domain of Schur decomposition. This binding makes the generator amenable to truncation and does not limit exploring all the possible modes. We slightly modify the BigGAN architecture incorporating residual network for synthesizing 2D representations of audio signals which enables reconstructing high quality sounds with some preserved phase information. Additionally, the proposed conditional training scenario makes a trade-off between fidelity and variety for the generated spectrograms. The experimental results on UrbanSound8k and ESC-50 environmental sound datasets and the Mozilla common voice dataset have shown that the proposed GAN configuration with the conditioning trick remarkably outperforms baseline architectures, according to three objective metrics: inception score, Frechet inception distance, and signal-to-noise ratio. 

Reviews and author responses:

We thank the reviewer for the valuable comments which helped us to improve our paper. We address all the raised concerns in detail as the following.

We sincerely thank the reviewer for sharing these two valuable references about cycle-consistent GANs. To deal with other aspects of GANs and speech, we have included these two references in Introduction of the revised manuscript, even if both references focus on a classification task. Our paper, on the other hand, does not focus on domain adaptation, improving classification accuracy, or cycle-consistent generative models. The scope of our paper is synthesizing audio/speech signals using GAN for the purpose of crafting high quality signals, and our objective is to improve the stability of the model during the training phase so that it better serves our purpose. More specifically, we bring attention to a common problem of GAN training (including cutting-edge GANs), which is instability and mode collapse. In response to this issue, we introduce the DDFN metric for correctly conditioning the generator network.

Answer-to-Comment-1:
Fig. 1 provides an example on how the DDFN can only indicate potential collapse for the baseline GAN architectures. As shown, doubling the batch size considerably delays collapse for the BigGAN architecture. However, Table 1 compares the results achieved for the different settings on our slightly modified BigGAN architecture. This table in the 1st submitted version reports:
1. #iteration which gives the highest achieved IS/FID before initial collapse
2. IS
3. FID

For clarity purposes, we have added the #iteration which points to extreme collapse to this table in the revised version of the manuscript. According to this table, larger batch sizes have a positive impact on both IS and FID scores which measure the quality and variety of the synthesized spectrograms compared to original samples. 

Answer-to-Comment-2:
In fact, the aim of Table 1 is not comparing the effect of spectral normalization. This table compares the effect of orthogonal regularization and batch size on the modified BigGAN architecture whose generator and discriminator are conditioned with the DDFN and spectral normalization, respectively. Regarding our response to the first comment of the reviewer, this table shows that running orthogonal regularization and doubling the batch size further delays complete collapse for the model, though they do not necessarily improve IS/FID scores before initial collapse. In the revised manuscript, we added four new records to this table aiming to better distinguish the DDFN impact for the model compared to orthogonal regularization.

Answer-to-Comment-3:
The resolution shown in each record of Table 2 is associated with the highest IS/FID scores we could achieve for the models. We excluded some records in this table to focus on the models with higher IS/FID scores and also save spaces in the body of the manuscript. However, we provide a complete comparison in Appendix B of the revised manuscript. Regarding Table 4 in the Appendix, both IS and FID scores are very competitive on resolutions 128 and 256. This table also indicates the positive impact of the DDFN metric on the BigGAN model and its variants. 

More importantly, we have not proposed any normalization approach. We, in fact, condition the generator with the DDFN metric computed in Schur decomposition domain to improve the model stability. The interpretation of the normality in computing the DFN metric (Eq. 5) is different from a conventional normalization operation. As explained in Section 3 of the manuscript, normality in Eq. 5 denotes only the Frobenius norm of each spectrogram (mean energy). This is fundamentally different from the spectral normalization approach (Miyato et al., 2018).

Answer-to-Comment-4:
As we discussed earlier, the scope and focus of our manuscript are completely different from the references mentioned by the reviewer, which introduce cyclic GANs for classification purposes. Besides, we have evaluated our conditioning trick on the cutting edge and complex GANs, namely SA-GAN and BigGAN with residual blocks.

More importantly, contrary to what was mentioned by the reviewer, we have never proposed any normalization approach. Instead, we have simply used spectral normalization in our experiments.

Answer-to-Comment-5:
Once again, we are not proposing any normalization technique in the manuscript. We propose a conditioning trick for the generator to not only improve its stability but also make a reasonable trade-off between sample quality and variety (See Figure 2). 

Furthermore, speech classification and speech synthesis are two distinct research scopes with different objectives. Although GANs can be used for conventional classification purposes, we do not generate high quality spectrograms to improve classification accuracy, but synthesize high quality spectrograms for crafting natural sounding signals.


Answer-to-Comment-6:
Please see in Appendix A.



We thank the reviewer for the valuable comments which helped us to improve our paper. We address all the raised concerns in detail as the following.

Our audio/speech signal synthesis approach is not limited only to spectrograms with “missing phase information”, but with “preserved original phase information”. This helps to synthesize signals with higher quality compared to phase approximation-based algorithms which often add audible noises to the signals. We discuss this issue later in response to other comments.

About comparing with other regularization techniques, we have compared our proposed deference departure from normality metric (DDFN) with two cutting-edge regularization techniques, namely orthogonal regularization and spectral normalization in the manuscript.

In Section 4.1 we have theoretically discussed the difference between the DDFN and orthogonal regularization. We mentioned that the DDFN does not manipulate weight vectors at each layer through backpropagation therefore it is computationally very fast. Moreover in the first version of the manuscript, we experimentally compared these two metrics on benchmarking datasets as shown in Table 1. However, we noticed it requires more clarification regarding the reviewer’s comment. Therefore, in the revised version of the manuscript, we updated this table with adding four more records (highlighted). According to this updated table, the DDFN outperforms the orthogonal regularization for the majority of the cases in terms of IS/FID scores and delaying potential collapse. 

In Introduction, we have explained the functionality of the spectral normalization on the discriminator network. This approach is fundamentally different from the DDFN (see Section 3.2 for the DDFN intuition). The importance of this normalization is shown in Table 1. However, Table 2 of the manuscript compares the performance explicitly of the spectral normalization with DDFN. As shown in the latter table, BigGAN with spectral normalization and DDFN outperforms BigGAN with spectral normalization only.

About spectrograms, WaveNet and MelGAN, there are some advantages in using spectrograms for audio/speech synthesis instead of time-domain approaches:
1. spectrograms have much lower dimensionality, they are less noisy, and provide more distinctive learning features,
2. advanced architectures developed in the computer vision domain can be generalized to spectrograms,
3. models require much less training parameters on spectrograms.   
Since our main interest in this paper is synthesizing high quality signals with less audible noises, we do not use any phase vector approximation algorithm. Instead of that, we use original phase vectors derived from real samples of the given datasets.

WaveNet is completely different from the proposed signal synthesis approach. More importantly, while our main contribution in this paper is improving the stability of GANs during training, WaveNet is not in the context of GANs. The main advantage of our approach over WaveNet is that it can make a reasonable trade-off between sample variety and quality via changing epsilon in DDFN (Eq. 8) and alpha (see Section 4.2 and Figure 2). This allows reconstructing signals with desired quality.

MelGAN is not a time-domain approach and similar to our approach, it works with spectrograms. Whereas our signal synthesis approach, MelGAN completely ignores phase information in signal reconstruction. This of course negatively affects the quality of the reconstructed signals and adds ambient noise to the signals. We highly recommend reviewing Appendix D of the manuscript and listening to the synthesized speech signals reconstructed from MelGAN and other phase approximation-based approaches.

About availability of phase information, it is true that preserving original phase information for reconstruction may not always be the case. However, we think that reconstructing signals with original phase vectors is currently the only reliable approach to reconstruct high-quality signals. As we discussed in the manuscript and in Appendix D, common phase approximation-based approaches often add audible noises to the signal which negatively affect their quality. 

We have conducted all our experiments on three comprehensive benchmarking datasets: ESC-50 and US8k (environmental sounds), and MCV (speech). Both datasets include varieties of signals, which according to our experiments, show sensitivity to inaccurate and distorted phase vectors. In fact, this was our main motivation for using original phase information instead of approximation-based approaches. 

More importantly, the task of a generator is producing spectrograms seamless to original ones available in the training set. As long as this is met, we can assign the phase information of an original sample to the synthesized spectrogram and correctly reconstruct the signal. This does not pose any phase mismatch issue.



We thank the reviewer for the valuable comments which helped us to improve our paper. We address all the raised concerns in detail as the following.

Answer-to-comment-1:
We have never mentioned in the manuscript that our proposed DDFN metric is for improving classification accuracy. The main goal in this manuscript is not generating high-fidelity spectrograms for achieving higher classification performance. The scope of this manuscript is audio/speech signal synthesis using generative adversarial network (GAN), and our focus is improving stability of the model during training. This helps to synthesize high quality signals for several purposes/applications, and classification can be one of them. Therefore, any discussion on the potential applications of generating high quality signals including classification is out of the scope of this paper.

We have only mentioned in Introduction that the highest recognition accuracy has been achieved by the DeepSpeech system (Hannun et al., 2014) on the frequency-magnitude representations to indicate the importance of crafting high quality spectrograms as a potential benefit/application for this system, but not as our goal. Even in the computer vision domain, fundamental GAN research (e.g., investigating instability issues) is distinct from GANs for improving classification accuracy. A good reference addressing this aspect is: Brock, Andrew, Jeff Donahue, and Karen Simonyan. "Large scale gan training for high fidelity natural image synthesis." arXiv preprint arXiv:1809.11096 (2018). 

Both in the body and appendices of the manuscript, we have provided comprehensive experiments on three benchmarking audio and speech datasets, namely ESC-50, US8K, and MCV. These are large datasets, which include varieties of samples with different lengths, utterances, levels of noise, etc. For measuring the effectiveness of the proposed synthesis approach on different GAN architectures, we have computed IS, FID, and SNR scores, which are common metrics in GAN analysis. Moreover, we have shown that conditioning the generator with the proposed DDFN metric improves model stability with further delaying collapse onsets.

For objective measuring the sample diversity we used FID score. This metric penalizes models with lower variety. Specifically, lower FID score can be interpreted as higher diversity in spectrogram synthesis. According to the tables, the proposed approach results in a moderately high sample variety. Moreover, we have included some synthesized signals into the supplementary material for subjective evaluations.

Answer-to-comment-2:
We have included some results followed by a brief discussion on image generation into Appendix A of the revised manuscript and supplementary material.

Answer-to-comment-3:
We revised the manuscript and we have included more explanations and clarified some concepts. Moreover, Figure 2 is better explained.

Answer-to-comment-4:
Distinguishing differences between spectrograms represented in RGB images is very difficult to the human eye. For example, two very similar spectrograms with pixel-level L2 distance of about 3.5 (small distortion), sound different after reconstruction, though the human eye does not recognize their differences. Regarding the locality of the distortion on the spectrogram, it can cause different side effects on the signal such as local clipping noises.

For clarity purposes we have added new synthesized signals into folder “Synthesized-Speech” of the revised supplementary material. More information about the contents of this folder is provided in Appendix D.


We thank the reviewer for the valuable comments which helped us to improve our manuscript. We address all the raised concerns in detail as the following.

The scope of the manuscript is audio/speech signal synthesis using generative adversarial network (GAN), and our main objective is to improve the stability of the model during the training phase. This helps to synthesize high quality signals, useful for various applications in speech processing. The state-of-the-art models in audio/speech recognition (e.g. DeepSpeech, Kaldi, etc.) use some type of 2D representation (spectrogram) instead of raw 1D signals because:
1. Spectrograms have much lower dimensionality;
2. They are less noisy;
3. They provide more distinctive learning features;
4. Several advanced architectures developed in the computer vision domain can be generalized to spectrograms;
5. Deep models require much less training parameters on spectrograms.   

Signal synthesis with spectrograms also requires phase matrices. Since our top priority in this paper is synthesizing high quality signals with less audible noises, we do not use any phase vector approximation algorithm. Instead of that, we use original phase vectors derived from real samples of the given datasets. Therefore for synthesizing high quality signals, we need to synthesize high quality spectrograms. Toward this aim and with respect to the fact that GANs often suffer from instability and mode collapse issues, we propose a new conditioning approach for the generator network (See the proposition and Eq. 8). We have experimentally proven the positive impact of this conditioning trick on the cutting-edge GAN architectures trained on spectrograms.

Answer-to-Comment-1:
The DDFN metric with its essential algebraic and recently improved statistical details is explained in Section 3 of the manuscript. The intuition behind this metric is correlating the probability distribution of the generator to real sample distribution. This regularizes the generator network to craft spectrograms within the manifolds of real samples which consequently bind its learning behavior and results to better stability.

The DFN metric (Eq. 5) computes a single real value for a spectrogram relative to its eigenvalues in the subspace of Schur decomposition. According to Eq. 5, the DFN measures the depth between the Frobenius norm of a spectrogram (mean energy) and the summation of all its possible eigenvalues (which encode structural components). This depth is in fact a single-value descriptor for a spectrogram representing its geometrical position in a manifold. Therefore, the more similar two spectrograms become, the closer their DFN values (descriptors) are expected so that they lie in the same manifold. In other words, minimizing over the DDFN metric (Eq. 8) forces the generator to produce spectrograms in compliance to the manifold of real spectrograms.

Furthermore, in the manuscript we have experimentally proven the effectiveness of employing DDFN metric on GAN architectures for three comprehensive audio (US8K, ESC-50) and speech (MCV) datasets (see tables, graphs, and discussions).

Constraining the generator using the DDFN metric has some advantages. First, unlike the spectral normalization, it does not manipulate weight vectors of the model. Therefore, it does not negatively affect the computational complexity of the model during backpropagation in each epoch. Second, we compute this metric for all the training samples and this helps to span all possible variety in the dataset. Third, the value for epsilon in Eq. 8 of the manuscript is justifiable according to the required quality for the spectrogram. This helps to make a better trade-off between samples variety and quality.

Answer-to-Comment-2:
We updated the manuscript with more detailed definitions for these symbols.

Answer-to-Comment-3:
With respect to the above-mentioned explanations provided in response to the first comment, we updated Section 3 of the manuscript (See section 3.2).

Answer-to-Comment-4:
The first step in training a GAN with DDFN is computing DFN (Eq. 5) for real and generated samples. However, we have explained how to fast approximate this metric in Section 3.2 of the manuscript. For real samples once we compute the DFN, but this operation is needed for all the generated samples. The second step is initializing the differentiable interval for Eq. 6. We empirically set the infimum and supremum to -1 and 1, respectively. However, this is tunable according to the performance of the model. The third step is to calculate Eq. 1 with the constraint of Eq. 8.

Due to space shortage in the body of the manuscript, we added these steps into the Appendix C and supplementary material.

Answer-to-Comment-5:
Yes, unfortunately, they are latex typos because the compiler has not recognized a latex tag. We updated these typos in the manuscript.

Answer-to-Comment-6:
We provided brief results and discussion on the CIFAR-10 dataset in Appendix A.



This paper proposes a conditioning method to train the GANs. The conditioning trick is based on the DFN (departure from normality) metric computed in the spectrogram domain of Schur decomposition to ensure the correlation between real and generated samples. Experiments were performed on a few public audio datasets, and the results suggest that imposing the proposed constraint on the generators not only delays the collapse in training but also yields the better reconstruction performance.

Pros:

+ An important issue of stable training of the GANs is addressed.

+ The proposed conditioning trick is novel and interesting in some sense.


Cons:

- The major concern about this paper is the lack of thorough experimentation to prove the usefulness of the proposed method. The experiments and evaluation don't seem to sufficiently support the main arguments. In particular, the authors put an emphasis on the importance of generating the high-fidelity audio spectrograms for higher classification accuracies. This was never supported by any experiments. The reconstruction of speech samples are not sufficient to back up the arguments. Furthermore, another advantage of the proposed method the authors state was its ability to generate the samples with high diversity. This too was not demonstrated with experiments.

- The authors state that the proposed method is generalizable but it should be confirmed through more experiments on the other domains. Otherwise, it is recommended that it must be explicitly stated that the proposed method applies to audio or speech, including the title.

- The paper is not easy to follow. Some terms and notations are not adequately explained. So are the descriptions of the figures. For example, in Figure 2, it is difficult to follow the tradeoff between the quality and the diversity according to different alpha, epsilon values.

- Speech samples in the supplementary material don't seem to match the corresponding DWT spectrograms and waveforms. There are some clipping noises and sudden silences in audio while the spectrograms and waveforms look almost identical to those of the original audio.

To summarize, the paper presents a novel and interesting idea to tackle the important problem in the field, but fail to provide the experimental evidences to support the idea. Therefore, I recommend the paper is not ready to be published in its current form.



Summary

this paper propose a constraint in GAN training, to improve generated samples fidelity and stabilize training. The proposed conditioning is based on limiting the generator from departing normality function of real samples, which is computed in the spectral domain of Schur decomposition. It is claimed that this conditioning will not limit the exploration of all modes of real data distribution



Missing references: 
GAN for unsupervised and semi supervised speech classification

1- Hosseini-Asl E, Zhou Y, Xiong C, Socher R. Augmented cyclic adversarial learning for low resource domain adaptation. ICLR 2018

2- Hosseini-Asl E, Zhou Y, Xiong C, Socher R. A multi-discriminator cyclegan for unsupervised non-parallel speech domain adaptation. INTERSPEECH 2018



Comments:

1- what is the relation of batch size to # iteration before collapse? in Table 1, on US8k, it shows larger batch using spectral normalization result in faster collapse, while this is reverse on ESC-50 dataset. Figure 1 is only shown for US8k dataset only.

2- If using spectral normalization stabilize training, the results in Table 1 indicate this is not always true with different batch size and different dataset. 

3- Table 2: there is no evaluation of SA-GAN and BigGAN on 256 resolution for MCV. Also no 128 resolution for all models on US8k dataset. what is the reason behind selecting some resolutions only. Is the proposed normalization only better on some selected resolutions?

4- [1] and [2] in missing references proposed multi-discriminator to provide more informative gradient to generator on spectrogram space. It would be also interesting to explore this in combination with proposed normalization

5- what is the impact of the proposed normalization on speech recognition performance? It would be very helpful to add classification metric, since the proposed approach is evaluated on class-conditional GAN. 

6- The evaluation on image generation is also required for the proposed normalization approach. 

The paper proposes a novel conditioning-based regularization of GAN training. While it has been known that tying up the generator and discriminator's latent variables can help stabilize gan training, the proposed method is based on a different theory, thus making the algorithm novel. The experimental results also support the authors' argument about the improved stability and results in terms of objective metrics.

One thing I wish the paper could improve is about the reason as to why the proposed method is superior to the existing regularization techniques, which is somewhat buried in the mathematical details. 

Another more serious issue is that the method is strictly limited to spectrogram reconstruction. I am not sure if this is the best practice when it comes to generate audio signals, compared against the other generative models that are based on time-domain approaches (e.g., WaveNet or MelGAN). More specifically, the authors proposed to preserve the original phase information from the real examples and reused it to reconstruct the time-domain signal. Given the generative nature of the system, I wonder if this option is always possible. The authors might as well compare to the alternative cases, such as estimating the phase from the magnitude spectrograms (e.g. Griffin-Lim) or by trying to reconstruct both real and imaginary spectrograms to recover the full complex spectrogram. Also, given that the signals (mostly environmental sound) used in the experiments are less sensitive to the phase distortion compared to the other kinds, such as speech and music, I believe that the phase mismatch issue wasn't thoroughly handled in this paper. 

Summary:
This paper introduces an additional objective to GAN training called “difference departure from normality” that is meant to encourage correlation between real and generated samples projected into some subspace. This is intended to prevent instability during training.  The technique is applied to audio spectrogram generation.


Reason for score:
Honestly I didn’t completely understand this paper, so I’m marking low confidence.  However, I do feel like the paper was quite low on clarity.  The technique wasn’t well motivated or concretely explained.  And I found it odd that it was applied to spectrogram generation exclusively rather than image generation which is the standard testbed for GANs algorithms.


Comments:
* This paper could really use some improved motivation (e.g, Why is DDFN a good metric? Why would it improve training stability?).
* Some of the terms and symbols need to be defined for clarity, e.g., p_g, p_r.
* Section 3 would be improved by the inclusion of intuitive explanations to help the reader (and me) understand why the DDFN metric makes sense and what it represents. 
* I would recommend including the steps of the overall learning algorithm for those who want to better understand the concrete steps involved in implementing the approach and how the DDFN objective is integrating into the standard GAN learning algorithm. 
* There’s some parameter matrices that show up as $`_g$ and $`_d$ (backticks) in Section 3. Are these latex typos?
* I would recommend testing the approach on common image datasets, since that is the typical testbed for generic GAN improvements. 


Meta-review: The paper proposes a trick for stabilizing GAN training and reports experiment results on spectrogram synthesis. All the reviewers rate the paper below the bar, citing various concerns, including a lack of clarity and unconvincing results. Several reviewers suggest conducting evaluations in the image domain as most of the GAN training techniques are proposed in the image domain. After consolidating the reviews and rebuttal, the area chair finds the reviewer's argument convincing and would not recommend acceptance of the paper.

Decision: withdrawn-rejected-submissions