Forward method call for weight training

username: Asgard
cooked: <p>My PyTorch method isn’t automatically calling the forward method.</p>
<p>I’m trying to embed my graph adjacency matrix by aggregating neighbours and combining them (similar to GraphSAGE)</p>
<p>An adjacency matrix is of size nXn and the embedding will be of size nXd where d&lt;n.</p>
<p>So, basically in my code, an adjacency matrix of a graph is fed as input for the purpose of embedding and the forward method should return the embedding.</p>
<pre><code class="lang-python">
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np

class Setting(nn.Module):
    
    def __init__(self, A):
        
        super(Setting, self).__init__()
        
        self.A = A
        self.X = np.array(np.sum(A, axis=1))
        self.feature_len = 10

        self.L = 3
        self.n = 10
        self.z_dim = 8
        
        self.h = torch.empty(self.L-1,self.n, self.z_dim)
        
        self.h0 = torch.from_numpy(self.X)
        
        W0 = nn.init.xavier_uniform_(torch.empty(self.feature_len, self.z_dim))
        
        self.h[0] = torch.empty(self.n, self.z_dim)

        for v in range(self.n):
            self.h[0][v] = F.relu(self.h0[v]*W0)
        
    def forward(self, A):
        
        d_u = np.empty([self.n, self.n])
        
        for v in range(self.n):
            d_u[v] = self.X[v]*self.A[v]

        d_u = torch.from_numpy(d_u)        
                
        h_n = torch.empty(self.L-1, self.n, self.z_dim)

        rnn = nn.GRUCell(self.z_dim, self.z_dim)

        H = F.normalize(self.h[0], p=2, dim=1)
        
        for l in range(0,self.L-1):
            
            h_n[l] = torch.mm(d_u, self.h[l-1].double())
            self.h[l] = rnn(self.h[l-1], h_n[l]).detach().float()
            self.h[l] = F.normalize(self.h[l], p=2, dim=1)
                        
            H = torch.max(H, self.h[l])
   
        return H

</code></pre>
<p>Now based on the value of H, I want to train the weight W0, by taking MSELoss of H and H1 (another already known embedding).</p>
<pre><code class="lang-python">net = Setting(A).double()
loss = nn.MSELoss()
optimizer = torch.optim.Adam(Setting.parameters(), lr=0.1)

for epoch in range(10):
    H = net(A)
    loss_calc = loss(H, H1)
    loss_calc.backward()
    optimizer.zero_grad()

print(H)
</code></pre>
<p>How to train the weight W0 based on this architecture? Any modifications in the code are highly appreciated.</p>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>To make <code>W0</code> trainable, you would have to define it as an <code>nn.Parameter</code>.<br>
Also, it should be used in the computation of the output of your model.<br>
Currently it seems you are only using <code>W0</code> in your <code>__init__</code> method to calculate <code>self.h</code>, which is then overwritten in the <code>forward</code> method, so I’m not sure how the computation graph should look like.</p>

username: Asgard
cooked: <p>So, changing the weight to nn.Parameter should work, right?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np

class Setting(nn.Module):
    
    def __init__(self, A):
        
        super(Setting, self).__init__()
        
        self.A = A
        self.X = np.array(np.sum(A, axis=1))
        self.feature_len = 10

        self.L = 4
        self.n = 10
        self.z_dim = 8
        
        self.h = torch.empty(self.L-1,self.n, self.z_dim)
        
        self.h0 = torch.from_numpy(self.X)
        
        W0 = nn.parameter.Parameter(torch.FloatTensor((self.feature_len, self.z_dim)))
        
        self.h[0] = torch.empty(self.n, self.z_dim)

        for v in range(self.n):
            self.h[0][v] = F.relu(self.h0[v]*W0)
        
    def forward(self, A):
        
        d_u = np.empty([self.n, self.n])
        
        for v in range(self.n):
            d_u[v] = self.X[v]*self.A[v]

        d_u = torch.from_numpy(d_u)        
                
        h_n = torch.empty(self.L-1, self.n, self.z_dim)

        rnn = nn.GRUCell(self.z_dim, self.z_dim)

        H = F.normalize(self.h[0], p=2, dim=1)
        
        for l in range(1,self.L-1):
            
            h_n[l] = torch.mm(d_u, self.h[l-1].double())
            self.h[l] = rnn(self.h[l-1], h_n[l]).detach().float()
            self.h[l] = F.normalize(self.h[l], p=2, dim=1)
            H = torch.max(H, self.h[l])
   
        return H

</code></pre>
<p>So, I’ve updated the weight as <code>Parameter</code>, but I’m unclear on how to add W0 in the forward method. Also, note that the limits of the <code>for</code> loop are changed so that there is no overwriting.</p>
<p>So, my question is as follows:<br>
I first calculate the h[0] using W and h0 (this is different from h[0]). Now further, I’ll use these values to calculate h[1], h[2], …, h[L-1]. Max value from each of this is updated is updated into H with</p>
<blockquote>
<p>H = torch.max(H, self.h[l])</p>
</blockquote>
<p>This returned value H is compared with H1 (another already known embedding).</p>
<p>So, to make H and H1 embeddings equal, weight W0 is basic the trainable parameter. How to do that and what are the modifications required in the code?</p>
<p>Thanks a lot for your time and help.</p>

username: ptrblck
cooked: <p>I’m still unsure how the code should work.<br>
You are currently creating <code>W0</code> with two values. Is this your use case or would you like to create a parameter with the shapes <code>[self.feature_len, self.z_dim]</code>? In the latter case you would need to use a factory method such as <code>torch.zeros</code>, <code>torch.randn</code> etc.</p>
<p>You should also register the parameter via <code>self.W0 = nn.Parameter(...)</code>, so that it’ll be returned in <code>model.parameters()</code>, which is the usual way to forward the parameters to an optimizer.</p>
<p>Once the code is running, you should check, if <code>model.W0.grad</code> gives you a valid gradient, as I’m currently unsure if the reassignments detach the computation graph.</p>

username: Asgard
cooked: <p>This is how the working of the code is to be:</p>
<ol>
<li>
<p>An adjacency matrix A of size nXn is passed into the constructor and X is the degree matrix of A (of size nX1)</p>
</li>
<li>
<p>Also, the values L, n and z_dim are initialized and tensor h0 copies the value from  X. Also, another tensor h (not h0; sorry for the similar confusing names) of size (L-1)X n X z_dim is initialized in</p>
</li>
</ol>
<pre><code class="lang-auto">for v in range(self.n):
            self.h[0][v] = F.relu(self.h0[v]*W0)
</code></pre>
<ol start="3">
<li>
<p>The trainable parameter is W0 which is to be trained is of the dimension 10X8</p>
</li>
<li>
<p>Now, in the forward method, all the values of h are set here:</p>
</li>
</ol>
<pre><code class="lang-auto">for l in range(1,self.L-1):
            
            h_n[l] = torch.mm(d_u, self.h[l-1].double())
            self.h[l] = rnn(self.h[l-1], h_n[l]).detach().float()
            self.h[l] = F.normalize(self.h[l], p=2, dim=1)
                        
            H = torch.max(H, self.h[l])
</code></pre>
<p>Note that when l=1, h[l-1] = h[0] which is set as:</p>
<pre><code class="lang-auto">for v in range(self.n):
            self.h[0][v] = F.relu(self.h0[v]*W0)
</code></pre>
<p>So ultimately, all values of h and H tensor are dependent on W0 weight which has to be trained. The forward method will return H tensor which will be compared with H1 (another pre-defined tensor) and difference between the two is to be minimised.</p>
<p>Please ask if there’s any clarification needed. Thanks for your time and help.</p>