How can l load my best model as a feature extractor/evaluator?

DeepLearner17: <p>Hello,</p>
<p>l have stored my best model where the network is as follow</p>
<p>net</p>
<pre><code class="lang-auto">My_Net(
  (cl1): Linear(in_features=25, out_features=6, bias=True)
  (cl2): Linear(in_features=60, out_features=16, bias=True)
  (fc1): Linear(in_features=16, out_features=120, bias=True)
  (fc2): Linear(in_features=120, out_features=84, bias=True)
  (fc3): Linear(in_features=84, out_features=10, bias=True)
)
</code></pre>
<p>To load my best model, l did the following :</p>
<pre><code class="lang-auto">my_best_model = torch.load('path to best model')
my_model.keys()
dict_keys(['epoch', 'state_dict', 'best_prec1', 'optimizer'])
</code></pre>
<p><strong>What l would like to do ?</strong></p>
<p>How can load and use my_best_model as :</p>
<ol>
<li>Feature extractor ?</li>
<li>Evaluator on new examples ?</li>
</ol>
<p>Thank you</p>

ptrblck: <p>You would have to load the <code>state_dict</code> and set it to <code>eval()</code>:</p>
<pre><code class="lang-auto">model = MyModel(...)
model.load_state_dict(my_model['state_dict'])
model.eval()
</code></pre>
<p>Now you can use it to evaluate new samples:</p>
<pre><code class="lang-auto">new_sample = ...
output = model(new_sample)
</code></pre>
<p>What do you mean by feature extractor?<br>
Would you like to get a certain intermediate layer output from your model?</p>

DeepLearner17: <p>Thank you for your reply. By feature extractor, l mean get ouptut from the last fully connected layer before softmax</p>

ptrblck: <p>You could return this activation together with the output in your <code>forward</code> method a use a <code>forward_hook</code> on the layer.</p>

DeepLearner17: <p>What if l would like to do that for an intermediate layer lets say the forth layer ?</p>

ptrblck: <p>I’ve created a small code snippet using a forward hook to store one activation from <code>fc2</code>:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.cl1 = nn.Linear(25, 60)
        self.cl2 = nn.Linear(60, 16)
        self.fc1 = nn.Linear(16, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
        
    def forward(self, x):
        x = F.relu(self.cl1(x))
        x = F.relu(self.cl2(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.log_softmax(self.fc3(x), dim=1)
        return x


activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook


model = MyModel()
model.fc2.register_forward_hook(get_activation('fc2'))
x = torch.randn(1, 25)
output = model(x)
print(activation['fc2'])
</code></pre>

DeepLearner17: <p>Thanks a lot. Awesome</p>

DeepLearner17: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
In your  small code snippet, we need to make x in Variable before feeding it to  the model.</p>
<pre><code class="lang-auto">x=Variable(x) 
output = model(x)
</code></pre>
<p>And <code>print(activation['fc2'])</code><br>
returns<br>
*** KeyError: ‘fc2’</p>

ptrblck: <p>If you are using PyTorch &lt; <code>0.4.0</code>, you have to wrap it into a <code>Variable</code>.<br>
The most recent stable version is <code>0.4.0</code> where <code>Variables</code> and <code>tensors</code> were merged.<br>
Have a look at the <a href="https://pytorch.org/2018/04/22/0_4_0-migration-guide.html">Migration Guide</a>.<br>
You’ll find the install instructions on the <a href="https://pytorch.org/">website</a>.</p>
<p>The <code>KeyError</code> is strange. Have you registered the activation with <code>get_activation('fc2')</code>?<br>
The name argument is used to store the activation in the dict, so I’m wondering, why the error happened.</p>

DeepLearner17: <p>Yes,<br>
l did</p>
<blockquote>
<p>model.fc2.register_forward_hook(get_activation(‘fc2’))</p>
</blockquote>

ptrblck: <p>Which PyTorch and Python version are you using?</p>

DeepLearner17: <p>print(torch.<strong>version</strong>)<br>
0.3.1.post3</p>
<p>and python3.6</p>

ptrblck: <p>I just installed an env with PyTorch <code>0.3.1.post3</code> and Python <code>3.6.5</code>.<br>
Since in this PyTorch version, <code>Variables</code> and <code>tensors</code> weren’t merged, you have to use:</p>
<pre><code>x = Variable(torch.randn(1, 25))
</code></pre>
<p>Your code should therefore throw an error when you try to run <code>output = model(x)</code>.<br>
Besides that, the code runs fine on my machine.</p>

DeepLearner17: <p>Hi back <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you for you help/ But l get stuck once again  and l’m confused. Here is my network architecture (variable <code>net</code>)</p>
<p>net<br>
ConvNet_LeNet5(<br>
(cl1): Linear(in_features=25, out_features=32, bias=True)<br>
(cl2): Linear(in_features=25, out_features=64, bias=True)<br>
(fc1): Linear(in_features=51200, out_features=100, bias=True)<br>
(fc2): Linear(in_features=100, out_features=4, bias=True)<br>
)</p>
<p>my best network is saved as follow :</p>
<pre><code class="lang-auto">        save_checkpoint({
            'epoch': int(epoch) + 1,
            'state_dict': net.state_dict(),
            'best_prec1': best_prec1,
            'optimizer': optimizer.state_dict(),
        }, is_best)
</code></pre>
<p>Before  loading the best model let’s try forward hook as you suggested</p>
<p><code>net.fc2.register_forward_hook(get_activation('fc2'))</code></p>
<p>works well.</p>
<p>But we need to load the best model as a feature extractor. I did that as follow :</p>
<p><code>my_best_model=net.load_state_dict(torch.load('model_best.pth.tar'))</code><br>
l get the following error :<br>
<code>*** KeyError: 'unexpected key "epoch" in state_dict'</code><br>
Then l loaded the best model  without specifying  <code>load_state_dict</code> as follow :</p>
<pre><code class="lang-auto">my_best_model = torch.load('model_best.pth.tar')
my_best_model.keys()
dict_keys(['epoch', 'state_dict', 'best_prec1', 'optimizer'])
my_best_model=my_model['state_dict']
my_best_model.keys()
odict_keys(['cl1.weight', 'cl1.bias', 'cl2.weight', 'cl2.bias', 'fc1.weight', 'fc1.bias', 'fc2.weight', 'fc2.bias'])
my_best_model=my_model['fc2.weight']
</code></pre>
<p>I’m not sure if it’s the correct way to load fc2 as a feature extractor (since l didn’t succeed to do that with hook). Please correct me.</p>
<h1>x is a test example</h1>
<p><code>output=my_best_model(x)</code><br>
It returns *** TypeError: ‘torch.cuda.FloatTensor’ object is not callable</p>
<p><strong>All what is needed is to fix</strong><br>
my_best_model=net.load_state_dict(torch.load(‘model_best.pth.tar’)) # error  *** KeyError: ‘unexpected key “epoch” in state_dict’<br>
and<br>
<code>my_best_model.fc2.register_forward_hook(get_activation('fc2'))</code></p>
<p>Thank you for your help <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Since you saved your echeckpoint as a dict, you will also load it as such.<br>
Therefore to get your <code>state_dict</code> you have to call <code>checkpoint['state_dict']</code> on it.</p>
<p>Also, if you would like to use the <code>fc2</code> as a feature extractor, you would have to restore your complete model and calculate the complete forward pass with your sample.</p>
<p>Why did the hook approach not work?</p>

DeepLearner17: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> l have an l update,</p>
<p>l hope that l fixed loading  state dic correctly. I did that as follow :</p>
<pre><code class="lang-auto">pretrained_dict=torch.load('model_best.pth.tar') 
pretrained_dict=pretrained_dict['state_dict']
</code></pre>
<p>then</p>
<pre><code class="lang-auto">net.load_state_dict(pretrained_dict)
net.fc2.register_forward_hook(get_activation('fc2'))
</code></pre>
<p>Until here it works fine without any error.<br>
Can you confirm to me that with this process l load the pretrained weigh (best model) of net and not the initial random weight of net (used to initialize the network ?)</p>
<p>Then l did the following :<br>
<code>output = net(test_x[0], coord_test[0], adj_test[0], L_test[0],lmax_test[0],mask_test[0])</code></p>
<p>and l got the following error :<br>
***** IndexError: invalid index to scalar variable.**</p>
<p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for you help</p>

ptrblck: <p>The loading looks fine!</p>
<p>It looks like you use a LeNet style architecture and try to call the forward pass on multiple inputs.<br>
Could you print the shape of <code>test_x</code>?<br>
Also, what are the other tensors? I assume <code>test_y</code> is the target.</p>

DeepLearner17: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, problem solved. batch_size=4 , so l need to forward 4 exemples.</p>
<p>I have a question for you if you don’t mind  :</p>
<p>now l want to extract features from layer fc1. However, when l apply :</p>
<p><code>net.fc1.register_forward_hook(get_activation('fc1'))</code></p>
<p>l get always the output of fc2.</p>
<pre><code class="lang-auto">net
ConvNet_LeNet5(
(cl1): Linear(in_features=25, out_features=32, bias=True)
(cl2): Linear(in_features=25, out_features=64, bias=True)
(fc1): Linear(in_features=51200, out_features=100, bias=True)
(fc2): Linear(in_features=100, out_features=40, bias=True)
)
</code></pre>
<p>when l apply</p>
<p><code>net.fc1.register_forward_hook(get_activation('fc1'))</code><br>
l’m supposed to vector of 100 features however l get 40.</p>
<p>Does you hook function</p>
<pre><code class="lang-auto">activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook
</code></pre>
<p>works only for the last layer fc2 ?</p>
<p>Thank you</p>

ptrblck: <p>I’m not sure, why you don’t get the right activations.<br>
The model definition looks a bit strange. If the model is designed to run as a sequential model, the shapes of the linear layers don’t match together.</p>
<p>In my <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6?u=ptrblck">code example</a> the hooks run fine. Could you check, if you used the same code logic?</p>

DeepLearner17: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>It’s not a problem for the shpae . l use x.view() to reshape. I’m working with graph ConvNet.</p>
<p>Here is my init() and forward() functions</p>
<pre><code class="lang-auto">class My_net(nn.Module):

    def __init__(self, net_parameters):

        print('My_net')

        super(My_net, self).__init__()
        np.random.seed(seed)

        D, CL1_F, CL1_K, FC1_F, FC2_F, p = net_parameters

       	# Conv layer 1 
        
        self.cl1 = nn.Linear(CL1_K, CL1_F,10)
        Fin = CL1_K;
        Fout = CL1_F;
        scale = np.sqrt(2.0 / (Fin + Fout))
        self.cl1.weight.data.uniform_(-scale, scale)
        self.cl1.bias.data.fill_(0.0)
        self.CL1_K = CL1_K;
        self.CL1_F = CL1_F;


        # Conv layer 1 
        self.cl2 = nn.Linear(CL2_K, CL2_F,10)
        Fin = CL2_K * CL1_F;
        Fout = CL2_F;
        scale = np.sqrt(2.0 / (Fin + Fout))
        self.cl2.weight.data.uniform_(-scale, scale)
        self.cl2.bias.data.fill_(0.0)
        self.CL2_K = CL2_K;
        self.CL2_F = CL2_F;



        # FC1
        self.fc1 = nn.Linear(FC1Fin, FC1_F,10)
        Fin = FC1Fin;
        Fout = FC1_F;
        scale = np.sqrt(2.0 / (Fin + Fout))
        self.fc1.weight.data.uniform_(-scale, scale)
        self.fc1.bias.data.fill_(0.0)
        self.FC1Fin = FC1Fin

        # FC2
        self.fc2 = nn.Linear(FC1_F, FC2_F,10)
        Fin = FC1_F;
        Fout = FC2_F;
        scale = np.sqrt(2.0 / (Fin + Fout))
        self.fc2.weight.data.uniform_(-scale, scale)
        self.fc2.bias.data.fill_(0.0)




    def forward(self, x, d, L, lmax, coord, adj_matrix,n_nodes):



        # conv layer 1
        x = conv_layer(x, self.cl1, L, lmax, self.CL1_F, self.CL1_K,n_nodes,masked=True)
        x = F.relu(x)


	# conv layer 2
        x = conv_layer(x, self.cl1, L, lmax, self.CL1_F, self.CL1_K,n_nodes,masked=True)
        x = F.relu(x)

	# Fully connected layer 1

        x=self.fc1(x)
        x = F.relu(x)




        # Fully connected layer 2

        x = self.fc2(x)
        x = F.relu(x)

        return x
</code></pre>