Generating Voice with a 1-dimensional DCGAN - Sensible or not?

Koro: <p>Hello Community,</p>
<p>I’m relativly new to Pytorch and wanted to try something out.<br>
My Idea was to generate a voice, using the DCGAN Example from the Tutorial,<br>
found on this website: <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" class="inline-onebox" rel="noopener nofollow ugc">Using the PyTorch C++ Frontend — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>
<p>So I changed the Generator and Discriminator to work with 1-dimensional Convolutions, since<br>
that makes more sense for Soundsequences. I created Functions to transform Tensors to Wave-Files (16bit/Mono/44100 Samples/Second) and vice versa using this: <a href="https://github.com/adamstark/AudioFile/" class="inline-onebox" rel="noopener nofollow ugc">GitHub - adamstark/AudioFile: A simple C++ library for reading and writing audio files.</a> Wavefile Converter from Adam Stark.</p>
<p>The functions are described in WaveKonverter1D.hpp :</p>
<pre><code class="lang-auto">#pragma once
#include &lt;torch/torch.h&gt;
#include &lt;vector&gt;

#include "AudioFile.h"

void Tensor2Wave(torch::Tensor source, std::string name, std::string destination, const int64_t length)
{
	std::stringstream ss;
	ss &lt;&lt; destination &lt;&lt; name;

	//We don't change source
	torch::Tensor data = source.view({ length, });
	auto acc = data.accessor&lt;float, 1&gt;();

	AudioFile&lt;float&gt; audioFile;
	audioFile.setBitDepth(16);
	audioFile.setSampleRate(44100);
	audioFile.setAudioBufferSize(1, length);
	for (int i = 0; i &lt; length; ++i)
	{
		audioFile.samples[0][i] = acc[i];
	}
	audioFile.save(ss.str(), AudioFileFormat::Wave);
}

torch::Tensor Wave2Tensor(std::string source, const int64_t length)
{
	//Vektor
	torch::Tensor output = torch::zeros({ length, });
	auto acc = output.accessor&lt;float, 1&gt;();

	AudioFile&lt;float&gt; audioFile;
	audioFile.load(source);

	int sampleRate = audioFile.getSampleRate();
	int bitDepth = audioFile.getBitDepth();

	int numSamples = audioFile.getNumSamplesPerChannel();
	double lengthInSeconds = audioFile.getLengthInSeconds();

	int numChannels = audioFile.getNumChannels();

	float* samples = new float[numSamples];

	if (length != numSamples &amp;&amp; numChannels != 1)
	{
		std::cout &lt;&lt; "Fehler, Channelmenge oder Länge des Tensors passt nicht" &lt;&lt; std::endl;
		std::cout &lt;&lt; "Either # of Channels or length of Tensor inapproriate" &lt;&lt; std::endl;
		return output;
	}

	for (int i = 0; i &lt; numSamples; i++)
	{
		samples[i] = audioFile.samples[0][i];
		acc[i] = samples[i];
	}

	delete[] samples;

	return output;
}

std::vector&lt;torch::Tensor&gt; Wave2TensorSplit(std::string source, const int64_t length)
{
	std::vector&lt;torch::Tensor&gt; result;
	AudioFile&lt;float&gt; audioFile;
	if (!audioFile.load(source))
	{
		std::cout &lt;&lt; "Fehler, kann Audiodatei nicht öffnen" &lt;&lt; std::endl;
		std::cout &lt;&lt; "Cannot open Audiofile" &lt;&lt; std::endl;
		return result;
	}
	//Get #Samples
	int sampleRate = audioFile.getSampleRate();
	int bitDepth = audioFile.getBitDepth();
	int numSamples = audioFile.getNumSamplesPerChannel();
	int numChannels = audioFile.getNumChannels();

	int numClips = numSamples / length;
	if (numClips &lt;= 0)
	{
		std::cout &lt;&lt; "Fehler, Audiodatei zu kurz" &lt;&lt; std::endl;
		std::cout &lt;&lt; "Audio too short" &lt;&lt; std::endl;
		return result;
	}
	if (numChannels != 1)
	{
		std::cout &lt;&lt; "Fehler, Audiodatei nicht Mono" &lt;&lt; std::endl;
		std::cout &lt;&lt; "Audio not Mono" &lt;&lt; std::endl;
		return result;
	}
	if (bitDepth != 16)
	{
		std::cout &lt;&lt; "Fehler, Audiodatei nicht in 16bit Auflösung" &lt;&lt; std::endl;
		std::cout &lt;&lt; "Audio not 16bit PCM" &lt;&lt; std::endl;
		return result;
	}
	if (sampleRate != 44100)
	{
		std::cout &lt;&lt; "Fehler, Sample Rate ist nicht 44100 Samples/Sekunde" &lt;&lt; std::endl;
		std::cout &lt;&lt; "Samplerate not  at 44100 Samples/Second" &lt;&lt; std::endl;
		return result;
	}
	//For all Soundclips
	for (int k = 0; k &lt; numClips; ++k)
	{
		//Create Tensor-Vector and add to std::vector 
		float* samples = new float[length];
		result.push_back(torch::empty({ length, }));
		auto acc = result[k].accessor&lt;float, 1&gt;();
		//Read one Audioclip per Tensor
		for (int i = 0; i &lt; length; ++i)
		{
			//Absolute Positon in Audiostream
			samples[i] = audioFile.samples[0][k * length + i];
			acc[i] = samples[i];
		}
		delete[] samples;
	}
	return result;
}
</code></pre>
<p>Then I create a DCGAN Model and train it with about 5 Minutes of me, reading a text as a .wav File.<br>
I also invert the amplitudes and added another 5 minutes of sound that way. I was hoping it might improve<br>
problems that the model could encounter when it comes to the phase of the soundwaves.</p>
<p>The actual DCGAN and the training-loop can be found in main.cpp :</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;
#include &lt;clocale&gt;

#include "WaveKonverter1D.hpp"
#include "AudioFile.h"

//Width of Output Tensor (1D)
const int64_t kLength = 150414; //About 3 - 4 Seconds of Sound at 44100 Samples/Second

//Noise Vektor
const int64_t kNoiseSize = 1000;

// After how many batches to create a new checkpoint periodically.
const int64_t kCheckpointEvery = 100;

//Batchsize
const int64_t kBatchSize = 5;

//Epochs
const int64_t kNumberOfEpochs = 2000; //Would take several days on GTX 980ti to finish

// After how many batches to log a new update with the loss value.
const int64_t kLogInterval = 1;

//Load Models + Optimizers from last session
const bool kRestoreFromCheckpoint = false;

class VoiceDataset : public torch::data::Dataset&lt;VoiceDataset&gt;
{
private:
	//Vector from 1D-Tensoren with each kLenght lenght
	std::vector&lt;torch::Tensor&gt; soundclips;

public:
	explicit VoiceDataset(const std::string&amp; soundclips_path, const int64_t length) : soundclips(Wave2TensorSplit(soundclips_path, length))
	{};
	//Override of Get-Method
	torch::data::Example&lt;&gt; get(size_t index) override;
	//Override of size-Method
	c10::optional&lt;size_t&gt; size() const override
	{
		return soundclips.size();
	}
};

torch::data::Example&lt;&gt; VoiceDataset::get(size_t index)
{
	//We ignore batch.target, we make softlabels
	return { soundclips[index], torch::ones({1,}) };
}


int main()
{
	//manuel seed? will it improve performance?
	torch::manual_seed(1);

	//Search for CUDA GPU
	torch::Device device(torch::kCPU);
	if (torch::cuda::is_available())
	{
		std::cout &lt;&lt; "CUDA is available! Training on GPU." &lt;&lt; std::endl;
		device = torch::Device(torch::kCUDA);
	}
	else
	{
		std::cout &lt;&lt; "Using CPU" &lt;&lt; std::endl;
	}

	//Generator from CNN + TCNN, big Kernelsizes for better performance?
	torch::nn::Sequential generator(
		//Layer 1
		torch::nn::Conv1d(torch::nn::Conv1dOptions(1, 32, 80).stride(2).padding(0).with_bias(false)),
		torch::nn::BatchNorm(32),
		torch::nn::Functional(torch::leaky_relu, 0.2),
		//Layer 2
		torch::nn::Conv1d(torch::nn::Conv1dOptions(32, 64, 96).stride(2).padding(3).with_bias(false).transposed(true)),
		torch::nn::BatchNorm(64),
		torch::nn::Functional(torch::relu),
		// Layer 3
		torch::nn::Conv1d(torch::nn::Conv1dOptions(64, 128, 96).stride(3).padding(3).with_bias(false).transposed(true)),
		torch::nn::BatchNorm(128),
		torch::nn::Functional(torch::relu),
		//Layer 4
		torch::nn::Conv1d(torch::nn::Conv1dOptions(128, 128, 64).stride(4).padding(2).with_bias(false).transposed(true)),
		torch::nn::BatchNorm(128),
		torch::nn::Functional(torch::relu),
		// Layer 5
		torch::nn::Conv1d(torch::nn::Conv1dOptions(128, 128, 32).stride(3).padding(1).with_bias(false).transposed(true)),
		torch::nn::BatchNorm(128),
		torch::nn::Functional(torch::relu),
		// Layer 6
		torch::nn::Conv1d(torch::nn::Conv1dOptions(128, 128, 16).stride(2).padding(0).with_bias(false)),
		torch::nn::BatchNorm(128),
		torch::nn::Functional(torch::leaky_relu, 0.2),
		// Layer 7
		torch::nn::Conv1d(torch::nn::Conv1dOptions(128, 128, 16).stride(2).padding(1).with_bias(false).transposed(true)),
		torch::nn::BatchNorm(128),
		torch::nn::Functional(torch::relu),
		// Layer 8
		torch::nn::Conv1d(torch::nn::Conv1dOptions(128, 64, 16).stride(2).padding(0).with_bias(false).transposed(true)),
		torch::nn::BatchNorm(64),
		torch::nn::Functional(torch::relu),
		// Layer 9
		torch::nn::Conv1d(torch::nn::Conv1dOptions(64, 1, 4).stride(2).padding(0).with_bias(false).transposed(true)),
		torch::nn::Functional(torch::tanh)
	);
	//To GPU
	generator-&gt;to(device);

	//Diskirminator with CNN + FFNN
	torch::nn::Sequential discriminator(
		// Layer 1
		torch::nn::Conv1d(
			torch::nn::Conv1dOptions(1, 64, 64).stride(1).padding(1).with_bias(false)),
		torch::nn::Functional(torch::leaky_relu, 0.2),
		// Layer 2
		torch::nn::Conv1d(
			torch::nn::Conv1dOptions(64, 128, 96).stride(3).padding(1).with_bias(false)),
		torch::nn::BatchNorm(128),
		torch::nn::Functional(torch::leaky_relu, 0.2),
		// Layer 3
		torch::nn::Conv1d(
			torch::nn::Conv1dOptions(128, 128, 96).stride(4).padding(1).with_bias(false)),
		torch::nn::BatchNorm(128),
		torch::nn::Functional(torch::leaky_relu, 0.2),
		// Layer 4
		torch::nn::Conv1d(
			torch::nn::Conv1dOptions(128, 128, 64).stride(5).padding(0).with_bias(false)),
		torch::nn::BatchNorm(128),
		torch::nn::Functional(torch::leaky_relu, 0.2),
		// Layer 5
		torch::nn::Conv1d(
			torch::nn::Conv1dOptions(128, 128, 32).stride(2).padding(0).with_bias(false)),
		torch::nn::BatchNorm(128),
		torch::nn::Functional(torch::leaky_relu, 0.2),
		//Layer 6
		torch::nn::Conv1d(
			torch::nn::Conv1dOptions(128, 1, 16).stride(1).padding(0).with_bias(false)), //1214
		torch::nn::Functional(torch::sigmoid),
		//Layer 7
		torch::nn::Linear(1214, 500),
		torch::nn::Functional(torch::sigmoid),
		//Layer 8
		torch::nn::Linear(500, 1),
		torch::nn::Functional(torch::sigmoid)
	);
	//To GPU
	discriminator-&gt;to(device);
	//Load Data (over 5 min of Recorded Speech + 5 min of recorded Speech with inverted Amplitudes)
	auto data_set = VoiceDataset("VoiceMe\\Recording.wav", kLength).map(torch::data::transforms::Stack&lt;&gt;());

	const int64_t batches_per_epoch =
		std::ceil(data_set.size().value() / static_cast&lt;double&gt;(kBatchSize));

	auto data_loader = torch::data::make_data_loader&lt;torch::data::samplers::SequentialSampler&gt;(
		std::move(data_set),
		kBatchSize);
	//Create Optimizer
	torch::optim::Adam generator_optimizer(
		generator-&gt;parameters(), torch::optim::AdamOptions(2e-3).beta1(0.5));
	torch::optim::Adam discriminator_optimizer(
		discriminator-&gt;parameters(), torch::optim::AdamOptions(2e-3).beta1(0.5));

	if (kRestoreFromCheckpoint)
	{
		torch::load(generator, "generator-checkpointX.pt");
		torch::load(generator_optimizer, "generator-optimizer-checkpointX.pt");
		torch::load(discriminator, "discriminator-checkpointX.pt");
		torch::load(discriminator_optimizer, "discriminator-optimizer-checkpointX.pt");
	}

	int64_t checkpoint_counter = 1;
	int64_t exports = 0;
	for (int64_t epoch = 1; epoch &lt;= kNumberOfEpochs; ++epoch)
	{
		int64_t batch_index = 0;
		for (torch::data::Example&lt;&gt;&amp; batch : *data_loader) {
			// Train discriminator with real voice
			discriminator-&gt;zero_grad();
			torch::Tensor real_voices = batch.data.to(device);
			//1 as Softlabel goal
			torch::Tensor real_labels =
				torch::empty(batch.data.size(0), device).uniform_(0.8, 1.0);
			//Discriminator test [kBatchsize,1,kLenght]
			torch::Tensor real_output = discriminator-&gt;forward(real_voices.view({ batch.data.size(0),1,kLength }));
			torch::Tensor d_loss_real =
				torch::binary_cross_entropy(real_output, real_labels);
			d_loss_real.backward();

			// Train discriminator with fake voice
			torch::Tensor noise =
				torch::randn({ batch.data.size(0), 1, kNoiseSize }, device);
			torch::Tensor fake_voices = generator-&gt;forward(noise);
			torch::Tensor fake_labels = torch::empty(batch.data.size(0), device).uniform_(0.0, 0.2);
			torch::Tensor fake_output = discriminator-&gt;forward(fake_voices.view({ batch.data.size(0),1,kLength }));
			torch::Tensor d_loss_fake =
				torch::binary_cross_entropy(fake_output, fake_labels);
			d_loss_fake.backward();

			torch::Tensor d_loss = d_loss_real + d_loss_fake;
			discriminator_optimizer.step();
			//Not needed on GPU anymore, delete to save Space on VRAM
			fake_voices.to(torch::kCPU);

			// Train generator.
			generator-&gt;zero_grad();
			fake_labels.fill_(1);
			//Must regenerate Fake Voice
			torch::Tensor fake_voices_new = generator-&gt;forward(noise);
			torch::Tensor fake_output_new = discriminator-&gt;forward(fake_voices_new.view({ batch.data.size(0),1,kLength }));
			torch::Tensor g_loss =
				torch::binary_cross_entropy(fake_output_new, fake_labels);
			g_loss.backward();
			generator_optimizer.step();

			exports++;

			if (batch_index % kLogInterval == 0) {
				std::printf(
					"\r[%2ld/%2ld][%3ld/%3ld] D_loss: %.4f | G_loss: %.4f",
					epoch,
					kNumberOfEpochs,
					++batch_index,
					batches_per_epoch,
					d_loss.item&lt;float&gt;(),
					g_loss.item&lt;float&gt;());
			}

			if (exports % kCheckpointEvery == 0) {
				// Checkpoint the model and optimizer state.
				torch::save(generator, "generator-checkpointX.pt");
				torch::save(generator_optimizer, "generator-optimizer-checkpointX.pt");
				torch::save(discriminator, "discriminator-checkpointX.pt");
				torch::save(
					discriminator_optimizer, "discriminator-optimizer-checkpointX.pt");
				// Sample the generator and save the voice
				torch::Tensor sample = generator-&gt;forward(torch::randn(
					{ 1, 1, kNoiseSize }, device));
				//Save a generated Voice Sample
				std::stringstream ss;
				ss &lt;&lt; "out" &lt;&lt; checkpoint_counter &lt;&lt; ".wav";
				Tensor2Wave(sample.to(torch::kCPU), ss.str(), "VoiceMe\\", kLength);
				std::cout &lt;&lt; std::endl &lt;&lt; "Resynthesis completed: " &lt;&lt; "VoiceMe\\" &lt;&lt; ss.str() &lt;&lt; std::endl;
				std::cout &lt;&lt; "\n-&gt; checkpoint " &lt;&lt; ++checkpoint_counter &lt;&lt; '\n';
			}
		}
	}

	std::cout &lt;&lt; "Training complete!" &lt;&lt; std::endl;

	return EXIT_SUCCESS;
}
</code></pre>
<p>As mentioned above, I’ve based it on the Code in the Tutorial. Now the problem I have is that<br>
after running this Code for 21 Hours, the results hardly sound any human to me. There’s always a monotone Frequency audible in the background, and the Waveforms don’t look similar to a natural voice:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/357af755a449e7996d73a037f32c504c47d493d1.png" data-download-href="https://discuss.pytorch.org/uploads/default/357af755a449e7996d73a037f32c504c47d493d1" title="2019-05-08"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/357af755a449e7996d73a037f32c504c47d493d1.png" alt="2019-05-08" data-base62-sha1="7D6L6gTI32PmQsyWukHWkMl031T" width="690" height="176" data-dominant-color="B2B2BF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2019-05-08</span><span class="informations">1204×308 12.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Above: My voice Below: Output after about 21 Hours of Training</p>
<p>What’s weird is, that the Wave is kind of clipping at the maximum.</p>
<p>Now I have two questions:</p>
<ol>
<li>Is there any way a DCGAN like the one I’m using could ever generate something that sounds like a human voice?</li>
<li>If so, what would you change here (Layers/Optimizers/Trainingdata/…)?</li>
</ol>
<p>I’m open for any suggestions and advice, since I don’t have much experience in deep learning. I also accept python code, I’m only using the C++ API here, because I’m more used to C++.</p>
<p>OS: Windows 10<br>
IDE: Visual Studio Community 2019<br>
GPU: GTX 980ti<br>
Libtorch: Latest Nightly Build for CUDA 10 (I don’t know how to read an accurate version string)</p>
<p>To Mention:<br>
The Discriminator Loss approaches a value of 0.7 very rapidly after the first checkpoint and stays there forever, while the Loss of the Generator stays at around 2.5. I’ve read that a discriminator loss, rapidly approaching 0 is a bad sign, does this apply here as well?<br>
I’ve also bumped up the Kernel sizes in both the Generator and Discriminator, hoping that it would help the networks to learn more complex patterns. Does that make sense? It clearly causes my GPU to require more VRAM, I’m almost using all of the 6GB at this point. So my GPU is at it’s limits with this model, that’s why I can only use a batchsize of 5.<br>
In the originial example, they’re using a manual seed of 1, is this important for successful learning?</p>
<p>If this Post is inappropriate for this category, since it’s a general problem I have, please tell me.</p>
<p>Thank you in advance for any help<br>
Regards,<br>
Florian</p>
<p>Update:<br>
Here are screenshots of my current training session (Checkpoint 1 is actually the 5th one, since I restarted the training)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f4a6df526008ca4d346afd47dd144dc63b3390ee.png" alt="2019-05-08%20(1)" data-base62-sha1="yUi1yGrfggy1OEgpoVw1XqvfGIC" width="476" height="480"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/2/2f5e6ddd1efc485aac20a09dbf9a19c611948f7b.png" alt="2019-05-08%20(2)" data-base62-sha1="6L2KMQCggVKSRh3FrxZZ9PuWyHp" width="461" height="482"><br>
You can see that the loss sticks at around these values</p>
<p>I’ve also uploaded a few output samples, so you can listen for youself, one is a snippet of the recording,<br>
the rest are results from the training. (Warning: High frequency, better turn down your volume!)<br>
Results: <a href="https://drive.google.com/open?id=1kIW-bCPP8LYXJSXplwp23J7-mm80unB5" class="inline-onebox" rel="noopener nofollow ugc">Results - Voice Generation - Google Drive</a></p>

dambo: <aside class="quote no-group" data-username="Koro" data-post="1" data-topic="44660">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/koro/48/16213_2.png" class="avatar"> Koro:</div>
<blockquote>
<p><code>explicit VoiceDataset(const std::string&amp; soundclips_path</code></p>
</blockquote>
</aside>
<p>Hello, I am trying something very similar. Were you able to come up with a working example?</p>

Koro: <p>Thanks for you interest,</p>
<p>I wasn’t really able to get satisfying results, however I’ve tried quite a few variants to get a working Text-to-Speech Model. I’ve abandoned the project a while ago, but the last Version I was trying didn’t use a GAN at all, I’ve actually settled on an Attention-Based LSTM with an Encoder + Decoder LSTM Network. But even then the attention was not really really learning to shift focus correctly, the model was suffering from overfitting. Trainnig Samples sounded very natural, but Validation Tests were very robotic, however you could slightly understand what it was saying. Back then I didn’t bother to go furhter as I was lacking enough knowledge.</p>
<p>If you are interessted in training a TTS Model, I recommend just checking out the existing Models like Tacotron+Waveglow or ForwardTacotron+WaveRNN/MelGAN, they sound pretty good.</p>
<p>If you still want to make your own TTS Model, I would highly recommend to split it up into a Mel-Spectogramm-Generator and a Vocoder as it makes it a lot easier to convert from short sequences like text to long sequences like Audio Samples.</p>
<p>Good luck with your project.</p>
<p>Kind Regards,<br>
Florian</p>