Rbfn: Getting same result in each epoch

szZzr: <p>Hi, i am a newbie guy… and try to train an rbf network… I used MNIST database. And pytorch framework… The results are the same in each epoch…</p>
<p>Like that…</p>
<pre><code class="lang-auto">Epoch: 1  
Accuracy: 0.815 	Loss: 5.701 	Recall: 0.507 	Precision: 0.340


Epoch: 2  
Accuracy: 0.815 	Loss: 5.628 	Recall: 0.507 	Precision: 0.340


Epoch: 3  
Accuracy: 0.815 	Loss: 5.570 	Recall: 0.507 	Precision: 0.340


Epoch: 4  
Accuracy: 0.815 	Loss: 5.523 	Recall: 0.507 	Precision: 0.340


Epoch: 5  
Accuracy: 0.815 	Loss: 5.486 	Recall: 0.507 	Precision: 0.340


Epoch: 6  
Accuracy: 0.815 	Loss: 5.456 	Recall: 0.507 	Precision: 0.340

</code></pre>
<p>and that’s happens with several rbf settings… i m change the way of centers inits, sigma inits, number of clusters and batch size, learning rate… but still the same… it used to repeat the result of the first epoch following epochs, sometimes the loss changes just… like above</p>
<p>look my code:</p>
<pre><code class="lang-auto">class RBF(nn.Module):
    def __init__(self, in_layers, centers, sigmas):
        super(RBF, self).__init__()
        self.in_layers = in_layers[0]
        self.centers = nn.Parameter(centers)
        self.dists = nn.Parameter(torch.ones(1,centers.size(0)))
        # self.linear0 = nn.Linear(in_layers[0], in_layers[0], bias = True)
        self.linear1 = nn.Linear(centers.size(0), in_layers[1], bias = True)

    def forward(self, x):
        phi = self.radial_basis(x)
        out = torch.sigmoid(self.linear1(phi.float()))
        return out

    def radial_basis(self,x):
        c = self.centers.view(self.centers.size(0),-1).repeat(x.size(0), 1, 1)
        x = x.view(x.size(0),-1).unsqueeze(1).repeat(1, self.centers.size(0),1)
        phi = torch.exp(-self.dists.mul((c-x).pow(2).sum(2, keepdim=False).pow(0.5) ))
        return phi

</code></pre>
<p>i have try and this radial_basis with same results:</p>
<pre><code class="lang-auto">def radial_basis(self,x):
        x = x.view(x.size(0),-1)
        size = [self.centers.size(0), x.size(0)]
        sigma = self.sigmas
        dists = torch.empty(size).to(device)
        for i,c in enumerate(self.centers):
            c = c.reshape(-1,c.size(0))
            temp = (x-c).pow(2).sum(-1).pow(0.5)
            dists[i] = temp
        dists = dists.permute(1,0)
        phi = torch.exp(-1*(dists/(2*sigma))) #gaussian
        return phi
</code></pre>
<p>and the training method below:</p>
<pre><code class="lang-auto">def training(engine, batch, device, model, criterion, optimizer):
    inputs, labels = batch[0].to(device), batch[1].to(device)
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
    return outputs, labels
</code></pre>
<p>i m not sure if it’s architecture’s problem… or something with the weights… it’s like backward() function not working…</p>

albanD: <p>Hi,</p>
<p>Your loss seems to be coming down, but very very slowly. Have you tried increasing the learning rate to speed it up?</p>

szZzr: <p>yes i have try from 1–&gt;0.001<br>
but nothing happens… i have some examples in which the loss still the same…</p>
<p>maybe some parameters aren’t define with the right way???</p>
<p>it estimates the centres with kmeans but i have try with randoms centers and gaussian distribution initialise centers…!!! but the same results</p>

albanD: <p>But the loss is going down? It is just that the accuracy did not change yet.</p>

szZzr: <p>yeah! the accurasy and the other metrics like precision and recall still the same!<br>
the loss has very little difference! You can check the result below:<br>
these are the settings</p>
<pre><code class="lang-auto">    batch = 10
    classes = ('0', '1', '2', '3', '4', '5', '6', '7', '8', '9')
    dim = 28*28
    learning_rate = 1
    epochs = 14
    clusters = 10
    executions = 0
    momentum = 0.9
</code></pre>
<p>ad the results</p>
<pre><code class="lang-auto">Epoch: 1  
Accuracy: 0.815 	Loss: 5.280 	Recall: 0.507 	Precision: 0.340


Epoch: 2  
Accuracy: 0.815 	Loss: 5.276 	Recall: 0.507 	Precision: 0.340


Epoch: 3  
Accuracy: 0.815 	Loss: 5.274 	Recall: 0.507 	Precision: 0.340


Epoch: 4  
Accuracy: 0.815 	Loss: 5.274 	Recall: 0.507 	Precision: 0.340


Epoch: 5  
Accuracy: 0.815 	Loss: 5.273 	Recall: 0.507 	Precision: 0.340


Epoch: 6  
Accuracy: 0.815 	Loss: 5.273 	Recall: 0.507 	Precision: 0.340


Epoch: 7  
Accuracy: 0.815 	Loss: 5.273 	Recall: 0.507 	Precision: 0.340
</code></pre>
<p>Do you think that this is normaL?</p>

albanD: <p>Given how small the change in loss is, I am not that suprise that the other metrics do not move.<br>
You want to find out why is your loss changing so slowly.<br>
Have you tried higher lr like 100 or 1000 ? You can increase it until your model actually diverges.</p>

szZzr: <p>i have the convection that the model is re-initialising, but i don’t know how…<br>
look this with lr = 100</p>
<pre><code class="lang-auto">Device: cuda:0
/usr/local/lib/python3.6/dist-packages/torchvision/datasets/mnist.py:53: UserWarning: train_data has been renamed data
  warnings.warn("train_data has been renamed data")
Epoch: 1  
Accuracy: 0.813 	Loss: 5.271 	Recall: 0.000 	Precision: 0.000


Epoch: 2  
Accuracy: 0.813 	Loss: 5.271 	Recall: 0.000 	Precision: 0.000


Epoch: 3  
Accuracy: 0.813 	Loss: 5.271 	Recall: 0.000 	Precision: 0.000


Epoch: 4  
Accuracy: 0.813 	Loss: 5.271 	Recall: 0.000 	Precision: 0.000


Epoch: 5  
Accuracy: 0.813 	Loss: 5.271 	Recall: 0.000 	Precision: 0.000
</code></pre>
<p>exactly the same loss, seems likes that something is going baaad</p>
<p>and just now, i try this with the lr=1000 i have the same resutls with lr=100 and lr=2… without a bit of differnce</p>

albanD: <p>Could you give a small (40/50 lines) code sample that reproduces this that I can run locally?</p>

szZzr: <p>i just commit all the code in my github to check plz!</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/szZzr/pytorch-RBFnet" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/22839494?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/szZzr/pytorch-RBFnet" target="_blank" rel="nofollow noopener">szZzr/pytorch-RBFnet</a></h3>

<p>Contribute to szZzr/pytorch-RBFnet development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>it’s ok for you?</p>