Sparse network doesnt improve

username: TUM
cooked: <p>Hi there,</p>
<p>I want to implement a class of sparse neural networks, the picture is an example for the class H_0 for the parameters d = 5, d_star = 1, M_star = 2.</p>
<p>This network class works kind of good, but when I use a random uniform distributed values as input, I get always an almost constant output (untrained).</p>
<p>When I train the network, it gets better sometimes (if I use y_train = f(x_train) as labels for some nonlinear function f).</p>
<p>Unfortunally I want to use this network class, to implement an even deeper network class which uses H_0 networks as  its layers.<br>
Because the H_0 give almost constant output, the output of my “bigger” class is completely constant even after training.<br>
Any ideas why the networks behave like that?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76c180b8eb4dbdc02c0804be64dd3cb23c61b282.png" alt="Download" data-base62-sha1="gWyV09TiGxEGPSKcsbYbWbDYYuu" width="593" height="349"></p>
<p>This is my class, which uses the class smallDense to obtain the sparsity.</p>
<pre><code class="lang-auto">`
import math
import numpy as np

import torch
import torch.nn as nn
import torch.nn.functional as F

class smallDense(nn.Module):
  def __init__(self, d, d_star):
    super(smallDense, self).__init__()
    self.fc1 = nn.Linear(d,4*d_star)
    self.fc2 = nn.Linear(4*d_star,1)
  
  def forward(self, input):
    x = torch.sigmoid(self.fc1(input))
    x = torch.sigmoid(self.fc2(x))
    return x
    

class H_0(nn.Module):
  def __init__(self, d, d_star, M_star):
    super(H_0, self).__init__()
    self.networks = nn.ModuleList([smallDense(d, d_star) for i in range(M_star)])
    
    self.fc_out = nn.Linear(M_star, 1)


  def forward(self, input):
    outputs = []
    for i in range(len(self.networks)):
      x = torch.sigmoid(self.networks[i](input))
      outputs.append(x)

    result = torch.cat(outputs, dim=1)
    
    x = self.fc_out(result)
    return x

net = H_0(5,1,3)

x_train = torch.rand(100,5)

net(x_train)`
</code></pre>

username: ptrblck
cooked: <p>In <code>H_0</code> you are applying <code>torch.sigmoid</code> on <code>self.networks[i](input)</code>, which has already applied a <code>sigmoid</code> on the output, so you are squashing the output again, which might be unwanted.</p>
<p>Could you check it, please?</p>
<p>Besides that I can’t find any obvious issues, but can neither comment if this type of cell would work, so let us know how your experiments worked out. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: TUM
cooked: <p>Oh, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>! This was unwanted, but unfortunally it doesn’t really behave differently after changing that.</p>
<p>What I don’t understand is, that the network gives me always outputs close to each other, even though the input differs (alot).</p>
<p>For example this input:</p>
<pre><code class="lang-auto">net1 = H_0(7,3,2)
net2 = H_0(7,3,2)

x_train = torch.tensor([[-500.0,-600.0,-340,0,-900000,-200,-66]])
x_train1 = torch.ones(1,7)
x_train2 = torch.rand(1,7)

print(net1(x_train))
print(net1(x_train1))
print(net1(x_train2))

print(net2(x_train))
print(net2(x_train1))
print(net2(x_train2))
</code></pre>
<p>results in the following output:</p>
<pre><code class="lang-auto">tensor([[-0.0762858838]], grad_fn=&lt;AddmmBackward&gt;)
tensor([[-0.0739296228]], grad_fn=&lt;AddmmBackward&gt;)
tensor([[-0.0765117779]], grad_fn=&lt;AddmmBackward&gt;)
tensor([[0.4121777117]], grad_fn=&lt;AddmmBackward&gt;)
tensor([[0.3294166028]], grad_fn=&lt;AddmmBackward&gt;)
tensor([[0.3289835751]], grad_fn=&lt;AddmmBackward&gt;)
</code></pre>
<p>For each network the output is kind of constant while the input differs.<br>
I know, that the sigmoid squashes into the interval [0,1] but shouldn’t it differ a bit more?</p>
<p>And if I use the H_0 networks in my bigger network recursively I get outputs that converge to a constant because each H_0 behaves like that.</p>
<p>By the way this network is motivated by a theoretical result for rates of convergence in nonparametric regression using feedforward neural networks by Kohler and Bauer (2019)<br>
Bauer, Benedikt; Kohler, Michael. On deep learning as a remedy for the curse of dimensionality in nonparametric regression.<br>
Ann. Statist. 47 (2019), no. 4, 2261–2285. doi:10.1214/18-AOS1747. <a href="https://projecteuclid.org/euclid.aos/1558425645" rel="noopener nofollow ugc">https://projecteuclid.org/euclid.aos/1558425645</a></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="TUM" data-post="3" data-topic="99906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/b487fb/48.png" class="avatar"> TUM:</div>
<blockquote>
<p>I know, that the sigmoid squashes into the interval [0,1] but shouldn’t it differ a bit more?</p>
</blockquote>
</aside>
<p>Not necessarily and you could check all intermediates to have a look what is “happening” in the model:</p>
<pre data-code-wrap="python"><code class="lang-python">class smallDense(nn.Module):
  def __init__(self, d, d_star):
    super(smallDense, self).__init__()
    self.fc1 = nn.Linear(d,4*d_star)
    self.fc2 = nn.Linear(4*d_star,1)
  
  def forward(self, input):
    x = torch.sigmoid(self.fc1(input))
    print('sigmoid(smallDense.fc1(x)) ', x)
    x = torch.sigmoid(self.fc2(x))
    print('sigmoid(smallDense.fc2(x)) ', x)
    return x

net1 = H_0(7,3,2)
net2 = H_0(7,3,2)

x_train = torch.tensor([[-500.0,-600.0,-340,0,-900000,-200,-66]])
x_train1 = torch.ones(1,7)
x_train2 = torch.rand(1,7)

print(net1(x_train))

print(net1(x_train1))
print(net1(x_train2))

print(net2(x_train))
print(net2(x_train1))
print(net2(x_train2))
</code></pre>
<p>You can see that the first input of course saturates the output of <code>sigmoid(fc1)</code>. However, since <code>sigmoid</code> squashes it to <code>[0, 1]</code>, <code>sigmoid(fc2)</code> could have “similar” outputs for <code>x_train</code> as for <code>x_train1</code> and <code>x_train2</code> (in my run <code>tensor([[0.5298]]</code> vs. <code>tensor([[0.5521]]</code>).</p>
<p>That doesn’t mean that the model is not trainable and your model might output larger differences for smaller input changes after training.</p>

username: TUM
cooked: <p>Okay, that helped alot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> thank you! Training progress also works fine now on the bigger model!</p>