Torch::nn::module::foward Procudes very similar results irrigardless of input

username: Kallinteris-Andreas
cooked: <p>My Module:</p>
<pre><code class="lang-auto">struct Net : torch::nn::Module {
	Net(int numIn, int numOut, int numHid, const size_t hid_count=1) {
		assert(hid_count &gt; 0);
		first = register_parameter("inputW", torch::rand({numIn, numHid}))/numHid;
		middle = new torch::Tensor[hid_count];
		h_c = hid_count;
		for (int i = 1; i != hid_count; i++)
			middle[i] = register_parameter("hidW"+std::to_string(i), torch::rand({numHid, numHid}))/numHid;
		last = register_parameter("outputW", torch::rand({numHid, numOut}))/numOut;
	}

	torch::Tensor forward(torch::Tensor input) {
		torch::Tensor output_layer,h;
		h = torch::sigmoid(torch::mm(input, first));
		for (int i = 1; i != h_c; i++)
			h = torch::sigmoid(torch::mm(h, middle[i]));
		output_layer = torch::sigmoid(torch::mm(h, last));
		return output_layer;
	}
	torch::Tensor first, last, *middle;
	size_t h_c;
};
</code></pre>
<p>my unit test:</p>
<pre><code class="lang-auto">test_foward(){
	const int input_nodes = 10, output_nodes=1, hidden_count = 1;
	Net nn (input_nodes, output_nodes, hidden_count);

	std::cout &lt;&lt; "Number of hidden layers: " &lt;&lt; hidden_count &lt;&lt; std::endl;
	for (int i = 0; i != 20; i++)
		std::cout &lt;&lt; nn.forward(torch::rand({output_nodes,input_nodes})).item&lt;float&gt;() &lt;&lt; std::endl;
}
</code></pre>
<p>sample terminal output</p>
<pre><code class="lang-auto">Number of hidden layers: 2
0.570727
0.570719
0.570772
0.570827
0.570737
0.570785
0.570691
0.570735
0.570691
0.570765
0.570737
0.57077
0.570822
0.570722
0.570809
0.570712
0.570836
0.570773
0.570783
0.570827
</code></pre>
<pre><code class="lang-auto">Number of hidden layers: 11
0.965095
0.965136
0.965416
0.964702
0.964923
0.96534
0.964738
0.965263
0.965538
0.965113
0.965482
0.965526
0.965461
0.965383
0.965067
0.964979
0.965134
0.965376
0.965572
0.96514
</code></pre>
<pre><code class="lang-auto">Number of hidden layers: 51
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
</code></pre>
<p>The title and the sample Outputs should illustrate the problem problem, if not please ask for  clarification</p>
<ul>
<li>Changing the activation faction does not remove the problem (it just changes the bounds of the output)</li>
<li>Removing the hidden Layer does not help</li>
<li>The Nature of the Input does not matter even all 0s produces the same result</li>
<li>Same for all 1s</li>
<li>
<pre><code class="lang-auto"> torch::Tensor fir = torch::rand({10, 5});
 for (int i = 0; i != 15; i++)//this works fine
 	std::cout &lt;&lt; torch::mm(fir, torch::rand({5,1})) &lt;&lt; std::endl;
</code></pre>
</li>
</ul>
<pre><code class="lang-auto">
Me and my project partener have spent hours on this issue, are we missing something or is there a bug
Note: you can answer if you do not know C++ anything helps</code></pre>

username: ptrblck
cooked: <p>You are initializing all parameters and the inputs with <code>torch::rand</code>, which will sample values from a uniform distribution on the interval <code>[0, 1)</code> and will thus yield only positive numbers.<br>
Increasing the layer size will thus easily saturate the activation and thus yield these saturated values.<br>
Use e.g. <code>torch::randn</code> to sample values from a Normal distribution and rerun the code.</p>

username: Kallinteris-Andreas
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the reply, 
but that is not my issue, (i want numbers bounded to [0.1))
</p>
<p>
My issue is that irrigardless of the input the module::foward() gives the same(ish) output
</p>

username: Kallinteris-Andreas
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Hmm the variance appears to be much higher you were right<br>
Thanks man</p>
<pre><code class="lang-auto">Number of hidden layers: 2
0.626091
0.644468
0.631702
0.592448
0.654189
0.61956
0.615086
0.634765
0.616084
0.642732
0.667842
0.637472
0.633968
0.647593
0.674249
</code></pre>

username: Kallinteris-Andreas
cooked: <p>Though that does not explain why feeding the NN all zeros does not provide all zeroes on the output (shouldn’t 0*x=0 (where 0 and x are tensors))</p>
<pre><code class="lang-auto">	for (int i = 0; i != 15; i++)
		std::cout &lt;&lt; nn.forward(torch::zeros({output_nodes,input_nodes})).item&lt;float&gt;() &lt;&lt; std::endl;
</code></pre>
<pre><code class="lang-auto">Number of hidden layers: 1
0.534954
0.534954
0.534954
0.534954
0.534954
0.534954
0.534954
0.534954
0.534954
0.534954
0.534954
0.534954
0.534954
0.534954
0.534954
</code></pre>

username: ptrblck
cooked: <p>If you are assigning zeros to the <code>bias</code> tensors, then yes. Otherwise the output would be the <code>bias</code> of the first layer and would then act as the output activation.</p>

username: Kallinteris-Andreas
cooked: <pre><code class="lang-auto">Net(int numIn, int numOut, int numHid, const size_t hid_count=1) {
		first = register_parameter("inputW", torch::randn({numIn, numHid}))/numHid;
		middle = new torch::Tensor[hid_count-1];
		for (int i = 1; i != hid_count; i++)
			middle[i] = register_parameter("hidW"+std::to_string(i), torch::randn({numHid, numHid}))/numHid;
		last = register_parameter("outputW", torch::randn({numHid, numOut}))/numOut;
	}
</code></pre>
<p>When i am Initing the module (and in extent it’s underling parameters/tensors) I do not specify anything about biases, what is the default behavior</p>
<p>I have been searching in the docks the other and could find an answer</p>
<p>you were probably  reffering to the bias of sigmoid?</p>
<p>anyway i think i need to go back to drawing board after reading more</p>

username: Matej_Kompanek
cooked: <p>Even though output of the first layer is all zeros, sigmoid of 0 is 0.5, middle and last layers then work with non-zero values.</p>

username: ptrblck
cooked: <p>That’s another good point.</p>
<p><a class="mention" href="/u/kallinteris-andreas">@Kallinteris-Andreas</a> by default trainable layers (such as <code>nn.Linear</code>) use a <code>bias</code> parameter as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear">docs</a>.</p>