Loss does not improve on training

username: MadmanNero
cooked: <p>Hi,</p>
<p>I implemented a model which fails to learn. Loss calculated for every epoch is exactly the same and so is the sequence of losses over batches for every epoch. I have been fumbling around with this for a couple of days and apparently stumbled over the reason for this behaviour now.</p>
<p>As it seems all Tensors I use in my C++ code are from namespace <strong>at</strong> even though I am explicitly creating them with <strong>torch::Tensor …</strong> . Can anyone please explain why this is so and how to fix this?</p>
<p>I’m including &lt;torch/torch.h&gt; and no other PyTorch headers an I’m never creating any <strong>at::Tensor</strong>s.</p>
<p>I only noticed this because of a compilation error I introduced where the compiler among other things complained about some <strong>at::Tensor</strong>s which I didn’t realize I was using until then.</p>
<p>Any help would be greatly appreciated.</p>
<p>Edit 1:<br>
Calling <strong>requires_grad()</strong> on any Tensor in my code so far yields <strong>false</strong>. Guess it should not be that way…</p>
<p>Edit 2: Here’s some code from the training part, maybe it’s just obvious what I’m doing wrong…</p>
<pre><code class="lang-auto">torch::optim::SGD optimizer(classifier-&gt;parameters(), 0.001);
const unsigned int epochs = 10;
torch::Tensor loss, output, runningLoss;

for(unsigned int epoch=0; epoch&lt;epochs; epoch++) {
	runningLoss = torch::zeros(1);
			
	for(vector&lt;torch::data::Example&lt;&gt;&gt;&amp; batch : *dataLoader) {
		for(torch::data::Example&lt;torch::Tensor, torch::Tensor&gt;&amp; sample : batch) {
			optimizer.zero_grad();
			output = classifier-&gt;forward(sample.data);
			loss = torch::mse_loss(output, sample.target);
			runningLoss += loss;
			loss.backward();					
			optimizer.step();
		}
	}			
}
</code></pre>
<p>Edit 3:<br>
Maybe I’m just doing something wrong when passing the data to the functions…</p>

username: MadmanNero
cooked: <p>I played arount with the learning rate of the SGD optimizer setting it to values from 10000000 to 0.0000001 without any effect… bump</p>

username: MadmanNero
cooked: <p>I’ve build an executable example now, stripped of all unnecessary stuff. Basicly I’m trying to build an autoencoder but the model is not converging. Have quite the same model in Python working like a charm. I really do not see where I might be going wrong here… In the example prog I’m passing one Tensor to the autoencoder which I expect the network should be able to learn and reproduce after some number of iterations. Loss is constant though no matter how many iterations …</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;

using namespace std;

struct Autoencoder : torch::nn::Module {
	
	unsigned int inputSize;
	const unsigned int encodingSize = 35;
	
	// Network layers
	torch::nn::Linear encode1;
	torch::nn::Linear encode2;
	torch::nn::Linear encode3;
	torch::nn::Linear decode1;
	torch::nn::Linear decode2;
	torch::nn::Linear decode3;
	
	Autoencoder() = delete;
	
	Autoencoder(unsigned int inputs)
		: inputSize(inputs),
		  encode1(register_module("encode1", torch::nn::Linear(inputSize, 77))),
		  encode2(register_module("encode2", torch::nn::Linear(77, 44))),
		  encode3(register_module("encode3", torch::nn::Linear(44, encodingSize))),
		  decode1(register_module("decode1", torch::nn::Linear(encodingSize, 44))),
		  decode2(register_module("decode2", torch::nn::Linear(44, 77))),
		  decode3(register_module("decode3", torch::nn::Linear(77, inputSize)))
	{}
	
	torch::Tensor forward(torch::Tensor&amp; input) {
		torch::Tensor x;
		
		// Encode
		x = torch::relu(encode1-&gt;forward(input));
		x = torch::relu(encode2-&gt;forward(x));
		x = torch::relu(encode3-&gt;forward(x));
		
		// Decode
		x = torch::relu(decode1-&gt;forward(x));
		x = torch::relu(decode2-&gt;forward(x));
		x = decode3-&gt;forward(x);
		
		return x;
	}
};

int main(int argc, char* argv[]) {
	
	torch::Tensor data = torch::zeros(10, torch::requires_grad(true).dtype(torch::kFloat32));
	
	data[0] = 0;
	data[1] = 1;
	data[2] = 0;
	data[3] = 1;
	data[4] = 0;
	data[5] = 1;
	data[6] = 0;
	data[7] = 1;
	data[8] = 0;
	data[9] = 1;
	
	Autoencoder autoencoder(10);
	
	torch::Tensor loss = torch::zeros(1);
	torch::Tensor output = torch::zeros(10);
	torch::optim::SGD optimizer(autoencoder.parameters(), torch::optim::SGDOptions(10));
	
	for(int i=0; i&lt;1000000; i++) {
		optimizer.zero_grad();
		
		output = autoencoder.forward(data);
		
		loss = torch::mse_loss(output.detach(), data.detach());
		
		cout &lt;&lt; "\rLoss: " &lt;&lt; loss.item().toFloat();
		
		loss.backward();
		optimizer.step();
	}
	
	cout &lt;&lt; endl;
	
	return 0;
}
</code></pre>
<p>Edit 1: Tried to explicitly initialize layer. Didn’t help <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"><br>
Edit 2: Tried to use different optimizer (Adam). Didn’t help <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"><br>
Edit 3: Moved detach() out of forward() to make it more visible…</p>
<p>Why do I have to detach those two tensors when passing to loss function? Is that correct?</p>

username: tom
cooked: <aside class="quote no-group" data-username="MadmanNero" data-post="3" data-topic="52289">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/bbce88/48.png" class="avatar"> MadmanNero:</div>
<blockquote>
<p>Why do I have to detach those two tensors when passing to loss function? Is that correct?</p>
</blockquote>
</aside>
<p>No. That means no training. You actually want to detach data (if it requires gradient) but not output.</p>
<p>Best regards</p>
<p>Thomas</p>

username: MadmanNero
cooked: <p>Hi tom, thanks for your reply! This get’s my sample program working <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Will check my initial program to see if it has the same issue.</p>

username: MadmanNero
cooked: <p>I’m wondering if the following could be problematic…</p>
<pre><code class="lang-auto">for(auto&amp; sample : batch) {
    . . .
    output = classifier-&gt;forward(sample.data);
    loss = torch::mse_loss(output, sample.data.detach());
    . . .				
</code></pre>
<p>input and target for the network are the same tensor. So if I detach sample.data do I also implicitly detach output? Or does this in some other way mess up gradients / learning?</p>
<p>Edit: Sorry, forget this. The example prog does just the same and is working…</p>

username: MadmanNero
cooked: <p>In my initial program it makes no difference if I detach the output variable or not… <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"> Loss remains constant either way.</p>
<p>The code is as follows:</p>
<pre><code class="lang-auto">torch::Tensor output = torch::zeros(127, torch::requires_grad(true));
for(unsigned int epoch=0; epoch&lt;epochs; epoch++) {
	optimizer.zero_grad();			
	output = classifier-&gt;forward(data_x);
	loss = torch::mse_loss(output, data_x.detach_());
	loss.backward();
	optimizer.step();
}
</code></pre>
<p>Is it possible that the forward() method creates an output Tensor without gradient information?</p>

username: MadmanNero
cooked: <p>… yep. Had the same detach() in my forward method as in the example above before I moved it… Now my loss turns out to be -nan …<img src="https://discuss.pytorch.org/images/emoji/apple/weary.png?v=9" title=":weary:" class="emoji" alt=":weary:"></p>

username: MadmanNero
cooked: <p>The -nan values are introduced by the Linear layers. The first layer introduces a handfull of those nans and it is getting worse with data traversing through each additional layer.</p>
<p>Most of the input data consists of zeros with usually a hand full of values differing from zero. Input data seems fine otherwise. No nan values to be found there.</p>
<p>Any suggestions/ideas on this?</p>
<p>Edit 1: Turns out that some bias values in the first layer are nan … How comes?<br>
Edit 2: Bias values are fine upon initialization. Nans seem to appear when updating the network…<br>
Edit 3: After learning on a first batch of size 64 a couple of nans already appear in the layer’s bias and weight values among some impressively big negative numbers. I’m afraid bias values are overflowing … My input data has not been scaled/normalized yet. I’ll try that and see if it helps…</p>

username: tom
cooked: <p>Did you move the learning rate back to some non-ridiculius value?</p>

username: MadmanNero
cooked: <p>Thanks tom for your reply. The learning rate was indeed set to 1.0 which is a bit high I guess but maybe not yet rediculous. Switching the learning rate back to 0.01 alone is not enough as it seems because I’m still getting my nans then.<br>
While I have still trouble getting a proper standard deviation to work, scaling by only subtracting the mean from input data seems to mitigate the problem. So far the network loss is decreasing / the network is converging in the couple of tests I did with the current settings.</p>
<p>So the solution to the specific problem in my case seemed to be</p>
<ol>
<li>Do not detach output</li>
<li>Scale input data</li>
</ol>
<p>As for the standard deviation problem I will open another thread.</p>
<p>Thanks tom for your help!</p>
<p>Edit 1: Guess I was cheering too early once again… In my scaling method I am subtracting the input data’s mean from input and dividing by the input data’s standard deviation which happens to be broken. As soon as I remove the division by the broken std-Tensor I keep getting nans again <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"></p>
<p>Edit 2: Interestingly when printing the result of<br>
<code>torch::std(data, 0);</code><br>
I get</p>
<blockquote>
<p>[ Variable[CPUType]{} ]</p>
</blockquote>
<p>Which I interpret as an empty Tensor with no values (?). Nevertheless division by that Tensor seems to scale values of the data Tensor which seems to fix the nan issue… weird. Is this correct expected behaviour?</p>

username: tom
cooked: <p>No, that is a tensor with empty shape, ie a scalar.</p>
<p>Besteht regards</p>
<p>Thomas</p>

username: MadmanNero
cooked: <p>Ah great, thanks again! So it seems like I have to provide another number as the second parameter to the std() function to get the standard deviation for every column seperately. I’ll find out <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: MadmanNero
cooked: <p>Interestingly<br>
<code>torch::std(tensor, integer)</code><br>
does not call the function one would think it does but (at least in my case) instead calls<br>
<code>torch::std(Tensor t, bool b)</code><br>
because the provided integer value is implicitly converted to a bool. I think this could be considered a bug or a design flaw, since this behaviour is counter intuitive and I think there will be more people using this the way I did. To provide the dimension along which to calculate the std deviation one has to explicitly call <code>torch::std(tensor, integer, false)</code></p>