Custom Linear Regression Layer

Mathedelic: <p>I want to write a custom Linear/Quadratic regression function in Pytorch of the form-<br>
def model(x):<br>
pred = x @ W @ x.t() + x @ m + b<br>
return pred<br>
where M is an nxn matrix, m is an nx1 vector, and b is a scalar. The model function maps an n-dimensional vector to a scalar.</p>
<p>In the end, after training I also want to see the values of matrix W, vector m and scalar b.</p>
<p>I tried using the nn.Module to write my own version but wasn’t successful. Any help would be very appreciated in this direction.</p>

ptrblck: <p>Writing a custom model sounds like the right approach.<br>
I assume you’ve taken a look at <a href="https://pytorch.org/tutorials/beginner/examples_nn/polynomial_module.html">this</a> or <a href="https://pytorch.org/tutorials/beginner/introyt/modelsyt_tutorial.html">this</a> tutorial? If so, could you describe where you are stuck right now?</p>

Mathedelic: <p>First of all, thank you so much for the quick response.<br>
I tried writing this custom module as the solution to my problem</p>
<pre><code class="lang-auto">class scr_model(torch.nn.Module):
    def __init__(self):
        """
        In the constructor we instantiate four parameters and assign them as
        member parameters.
        """
        super().__init__()
        self.W = torch.nn.Parameter(torch.randn((26,26)))
        self.m = torch.nn.Parameter(torch.randn((26,1)))
        self.b = torch.nn.Parameter(torch.randn((1)))

        

    def forward(self, x):
        """
        In the forward function we accept a Tensor of input data and we must return
        a Tensor of output data. We can use Modules defined in the constructor as
        well as arbitrary operators on Tensors.
        """
        return self.b + self.m * x +  x *self.W * x.t()  
</code></pre>
<p>But when i try to test it using</p>
<pre><code class="lang-auto">for x,y in train_loader:
    preds = model(x)
    print("Prediction is :\n",preds.shape)
    print("\nActual targets is :\n",y.shape)
    break
</code></pre>
<p>it shows me that<br>
Prediction is :<br>
torch.Size([26, 26])</p>
<p>Actual targets is :<br>
torch.Size([1, 1])</p>

ptrblck: <p>In your code you are using the elementwise multiplication instead of the <code>matmul</code> operation from your previous post.<br>
Change this and also the order of <code>self.m</code> and <code>x</code> and the output would have a shape of <code>[1, 1]</code>:</p>
<pre><code class="lang-python">class scr_model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.W = torch.nn.Parameter(torch.randn((26,26)))
        self.m = torch.nn.Parameter(torch.randn((26,1)))
        self.b = torch.nn.Parameter(torch.randn((1)))

    def forward(self, x):
        return self.b + x @ self.m  +  x @ self.W @ x.t()  
    
    
model = scr_model()
x = torch.randn(1, 26)
out = model(x)
print(out.shape)
# torch.Size([1, 1])
</code></pre>
<p>Note however that the output shape depends on the batch size which still sounds wrong.</p>

Mathedelic: <p>Thank you so much Sir !!!</p>

Mathedelic: <p>I came back to this problem, that the output shape depends on the batch size, so when I change the batch size to 100, it gives me the warning<br>
<code>UserWarning: Using a target size (torch.Size([100, 1])) that is different to the input size (torch.Size([100, 100])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.   return F.mse_loss(input, target, reduction=self.reduction)</code><br>
How to overcome this issue?</p>

J_Johnson: <p>Not at a computer, but could try changing this:</p>
<pre><code class="lang-auto">self.b = torch.nn.Parameter(torch.randn((1, 1)))
</code></pre>
<p>Note the extra dim. Haven’t tested, yet, but guessing that will do the trick.</p>

Mathedelic: <p>Hey Johsnon, thanks for the effort mate but it did not work out.  I posted the entire code which on this link for you to check (<a href="https://discuss.pytorch.org/t/using-a-target-size-torch-size-100-1-that-is-different-to-the-input-size-torch-size-100-100-this-will-likely-lead-to-incorrect-results-due-to-broadcasting-please-ensure-they-have-the-same-size/172626" class="inline-onebox">Using a target size (torch.Size([100, 1])) that is different to the input size (torch.Size([100, 100])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size</a>). Thanks again for the help mate!!!</p>

ptrblck: <p>I don’t know how you would like to solve this issue as this operation:</p>
<pre><code class="lang-python">x @ self.W @ x.t() 
</code></pre>
<p>creates an output in the shape <code>[batch_size, batch_size]</code> as already mentioned.<br>
Could you explain what this operation is supposed to do and how you are interpreting it?</p>

J_Johnson: <p>Tried your code and I’m in the same camp as <a class="mention" href="/u/ptrblck">@ptrblck</a>. The motivation isn’t clear. Are you trying from some type of self attention?</p>

Mathedelic: <p>I want to do a simple Quadratic regression model, given by the equation in the image.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86975c6af6ab5a521683b5b69d6df48aee0ac16a.png" alt="Capture" data-base62-sha1="jcEgLOhDbrPTkr32vNWM29KO4Yy" width="425" height="87"><br>
Now to do this, the first term is given by x @ self.W @ x.t(), as square becomes transpose here. Similarly the second term in the equation bx is written as x @ self.m.</p>
<p>Now I want to know the values of W and m such that my input is correctly mapped to my output</p>

J_Johnson: <p>A quadratic would be:</p>
<pre><code class="lang-auto">Y=self.W*X**2 + X*self.M + self.b
</code></pre>
<p>Where upper case are matrices and lower case are vectors/scalars.</p>
<p>You could do the above and get effectively a learnable set of quadratic equations.</p>
<p>However, matrix multiplication is an entirely different “ball game”, so to speak. When we have some learnable vector space, it’s more like this (below variables will represent scalars):</p>
<pre><code class="lang-auto">a_11*x_1 + a_21*x_1 + a_31*x_1 + a_41*x_1 ... a_n1*x_1 + b_1
</code></pre>
<p>Each scalar of the input vector effectively goes through the above when entering a Linear layer. So if you want to try some quadratic version of that, in addition to the standard linear layer, you might try:</p>
<pre><code class="lang-auto">return x**2@self.W + x@self.m + self.b
</code></pre>
<p>Additionally, if using the above, should set:</p>
<pre><code class="lang-auto">self.W = torch.nn.Parameter(torch.randn((26,1)))</code></pre>

Mathedelic: <p>Hey Johnson! Thank you again for the reply mate. The problem is that I want W to be a matrix, there is a reason why I am doing x.T <span class="mention">@W</span> x, and that is that at the end of training I want to extract this W matrix, so I cannot initiate W to be a vector of size (25,1).</p>

J_Johnson: <p>It would be a matrix if your output size was larger than 1. But if you have an output size of 1, then it would need to be a vector, or you will get a size larger than 1.</p>

J_Johnson: <p>The <code>dim = 1</code> of W and m should be whatever output size you need. And same goes for your bias vector.</p>

J_Johnson: <p>Alternatively, you could just sum across dim=1, i.e. <code>(x.T @W x).sum(dim=1)</code>, but just note that approach cross-contaminates batchwise inputs; vs. parallelized operations. Not sure if that’s what you’re going for.</p>

Mathedelic: <p>When W is a matrix of dimension (25,25). and x is a vector of dimension (1,25). Then x@W is a vector of dimension (1,25). However, when we then do x@W@x.T then we get a scalar because dimension x.T is (25,1). and thus the end result has dimension 1, or a scalar</p>

J_Johnson: <p>If I understand you correctly, you want your formulation <code>return self.b + self.m * x +  x *self.W * x.t()</code> to work EXACTLY the same as it does for a batch size of 1, but with a batch size of &gt; 1. Am I missing anything?</p>
<p>If that’s the case:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

torch.manual_seed(0)
class scr_model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.W = torch.nn.Parameter(torch.randn((26,26)))
        self.m = torch.nn.Parameter(torch.randn((26,1)))
        self.b = torch.nn.Parameter(torch.randn((1)))

    def forward(self, x):
        return self.b + x @ self.m  +  x @ self.W @ x.t()

class scr_model2(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.W = torch.nn.Parameter(torch.randn((26,26)))
        self.m = torch.nn.Parameter(torch.randn((26,1)))
        self.b = torch.nn.Parameter(torch.randn((1,1)))

    def forward(self, x):
        return self.b + x @ self.m  +  ((x@self.W)*x).sum(1).unsqueeze(1)

model1=scr_model()
model2=scr_model2()

with torch.no_grad():
    model2.W.data=model1.W.data
    model2.m.data=model1.m.data
    model2.b.data=model1.b.data

x=torch.arange(26).view(1,-1)
w=torch.arange(26*3).view(3,-1)

x=model1(x.float())
w=model2(w.float())
print(x, w)
print(torch.equal(x, w[0:1,:]))
</code></pre>
<blockquote>
<p>tensor([[9108.7402]], grad_fn=) tensor([[  9108.7402],<br>
[ 69386.3047],<br>
[188471.6406]], grad_fn=)<br>
True</p>
</blockquote>