Trainable Variable in Loss Function and Matrix Multiplication

melgor: <p>Hi, could you check the example with trainable-variable? I was thinking about two cases:</p>
<ul>
<li>implementing a Linear layer without torch.nn.</li>
<li>implementing the Loss layer with learable parameter (ex. Center Loss where mean-center is trainable parameters).</li>
</ul>
<p>Such stuff are easy in TesnotFlow, but I’m not sure if I coded it correctly in PyTorch</p>
<p>Here is the example of my try of implementing the Linear-Layer. I’m not sure if I should implement “parameters()” this or other. Or that maye PyTorch have a function which gather all trainable variable at single list.<br>
<a href="http://pastebin.com/NkFZJkBW" class="onebox" target="_blank" rel="nofollow noopener">http://pastebin.com/NkFZJkBW</a></p>
<p>Here is my try of using learnable variable in loss-function. It seems to work nice, but I’m not sure if I used the pytorch function correctly (as adding additional variable to optimizer) and as I want to one Loss applied to final layer and second to intermediate layer<br>
<a href="http://pastebin.com/X35jEE54" class="onebox" target="_blank" rel="nofollow noopener">http://pastebin.com/X35jEE54</a></p>
<p>Could you check if the idea of using PyTorch here is correct? Or I should change sth here.</p>

fmassa: <p>Hi!</p>
<p>There are some issues with your first script:</p>
<ul>
<li>you need to define <code>linear1</code> and <code>linear2</code> as <code>nn.Parameter</code> instead of <code>Variable</code>, if not <code>model.parameters()</code> won’t see them</li>
<li>just for information, <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/_functions/linear.py#L10"><code>nn.Linear</code> implements it as <code>torch.mm(input, weight.t()</code></a>, as in torch7 BTW</li>
<li>you might want to divide the loss by the batch size, that’s what’s the default in pytorch</li>
</ul>
<p>At a first sight, the second example looks fine to me. But note that you can do <code>y.size(0)</code> instead of <code>y.size()[0]</code></p>

waitwaitforget: <p>I’ve implemented the same loss, however, the centers learned are nearly the same. I don’t know where is the problem.</p>

jxgu1016: <p>I have implemented the center loss and it looks correct. See this <a href="https://github.com/jxgu1016/MNIST_center_loss.pytorch" rel="nofollow noopener">repo</a> <a class="mention" href="/u/waitwaitforget">@waitwaitforget</a> <a class="mention" href="/u/melgor">@melgor</a></p>

CKboss: <p>A more easily implenment of center loss.</p>
<pre><code class="lang-python">class CenterLoss(nn.Module):

    def __init__(self,center_num,feature_dim):
        super(CenterLoss,self).__init__()
        self.center_num = center_num
        self.feature_dim = feature_dim
        self.center_features = nn.Parameter(torch.Tensor(self.center_num,self.feature_dim))
        nn.init.normal(self.center_features.data,mean=0,std=0.1)

    def forward(self,x,label):
        B = x.size()[0]
        center = torch.index_select(self.center_features,0,label)
        diff = x-center
        loss = diff.pow(2).sum() / B
        return loss
</code></pre>

lelegu: <p>I had the same issue. My parameters learned are the same. Did you solve it and how? Thanks!</p>