How to extract learned weights correctly

Entrophy: <p>I am trying to extract the weights from a linear layer, but they do not change during training, although error is dropping monotonously. Printing the weights’ sum, nothing happens - it stays constant.</p>
<p>I tried to extract the weights in two ways, but both do not work:</p>
<p><code>list(model.parameters())[0].data.numpy()</code></p>
<p>or</p>
<p><code>model.fc2.weight.data.numpy()</code></p>
<p>Here are the other code snippets:</p>
<pre><code>def train(epochs):
    model.train()
    for epoch in range(1, epochs+1):
        # Train on train set
        print(np.sum(model.fc2.weight.data.numpy()))
        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = Variable(data), Variable(data)
            optimizer.zero_grad()
            output = model(data)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()
</code></pre>
<p>and</p>
<pre><code># Define model
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(100, 80, bias=False)
        init.normal(self.fc1.weight, mean=0, std=1)
        self.fc2 = nn.Linear(80, 87)
        self.fc3 = nn.Linear(87, 94)
        self.fc4 = nn.Linear(94, 100)

    def forward(self, x):
         x = self.fc1(x)
         x = F.relu(self.fc2(x))
         x = F.relu(self.fc3(x))
         x = F.relu(self.fc4(x))
         return x</code></pre>

alexis-jacq: <p>I can’t repeat your error, my idea is that your problem comes from your data set:</p>
<pre><code class="lang-python">model = Net()
optimizer = optim.Adam(model.parameters(), 0.01)
criterion = nn.MSELoss()

def train(epochs):
    model.train()
    for epoch in range(1, epochs+1):
        # I dont have your data set:
        data = Variable(torch.rand(1,100))
        target = Variable(torch.ones(1,100))
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        s = torch.sum(model.fc2.weight.data)
        print(s)

train(100)
</code></pre>
<p>this gives</p>
<p>2.79897637193<br>
5.36990540938<br>
7.54833696394<br>
9.55001756023<br>
11.2473634735<br>
13.7405790995<br>
16.014156151<br>
18.0902192594<br>
20.0432560109<br>
22.2485624477<br>
24.2093355584<br>
25.687371045<br>
27.022266482<br>
…</p>

Entrophy: <p>Thanks for your help.<br>
I prepared a minimal working example of my code. Maybe there is something wrong, because I am new to Pytorch and do not something important. Maybe you can see anything?</p>
<pre><code>import numpy as np
import numpy.random as rnd

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data as data_utils
import torch.optim as optim
import torch.nn.init as init

#
# Return data
#
def sparse_data(N, k, num):
    X         = np.zeros((N, num))
    X[0:k,:]  = np.abs(rnd.normal(0, 1, size=(k, num)))
    idx_1     = rnd.sample(X.shape).argsort(axis=0)
    idx_2     = np.tile(np.arange(X.shape[1]), (X.shape[0], 1))
    return np.transpose(X[idx_1, idx_2])
    
#    
# Define model
#
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(100, 80, bias=False)
        init.normal(self.fc1.weight, 0, 1)
        self.fc2 = nn.Linear(80, 90)
        self.fc3 = nn.Linear(90, 100)

    def forward(self, x):
        x = self.fc1(x)
        x = F.sigmoid(self.fc2(x))
        x = F.sigmoid(self.fc3(x))
        return x
    
#    
# Prepare data
#
# &lt;-- sample x --&gt;
X_train = sparse_data(N=100, k=20, num=500)
train = data_utils.TensorDataset(torch.from_numpy(X_train).float(), torch.from_numpy(X_train).float())
train_loader = data_utils.DataLoader(train, batch_size=1)

model     = Net()
optimizer = optim.SGD(model.parameters(), lr=0.01)
criterion = nn.MSELoss(size_average=False)

#
# Train
#
model.train()
init_weights = model.fc1.weight
for epoch in range(1, 3):
    for batch_idx, (dat, target) in enumerate(train_loader):
        data, target = Variable(dat), Variable(target)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
trained_weights = model.fc1.weight

#
# Output
#
print(torch.sum(trained_weights), torch.sum(init_weights), torch.sum(trained_weights-init_weights))
</code></pre>
<p>Output is:</p>
<pre><code>Variable containing:
 80.6124
[torch.FloatTensor of size 1]
 Variable containing:
 80.6124
[torch.FloatTensor of size 1]
 Variable containing:
 0
[torch.FloatTensor of size 1]</code></pre>

alexis-jacq: <p>I see. you are comparing two times two variables pointing to the same memory storage. Of course, you have zero difference.</p>
<p>Try this:</p>
<pre><code class="lang-python">import copy
init_weights = copy.deepcopy(model.fc1.weight.data)
for epoch in range(1, 3):
    for batch_idx, (dat, target) in enumerate(train_loader):
        data, target = Variable(dat), Variable(target).detach()
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        print(torch.sum(model.fc1.weight.data))
trained_weights = model.fc1.weight.data
</code></pre>

Entrophy: <p>Thanks! That worked well.<br>
Did not know this Python behaviour yet.<br>
Looking at the new results, I guess the “detach()” function is not neccessary!?</p>

kazem: <p>Your method doesn’t really work!</p>

thaara_pillai: <p>Hi,</p>
<p>Why are you doing torch.sum(model.fc1.weight.data) on the weights. This leads to an addition of all feature weights. The main purpose of extracting weight is to identify which feature is important for the task. Could you explain if I am interpreting it wrong?</p>