What should the shape of weights be here?

username: jpj
cooked: <p>I have a fully connected neural network with 78 input features and two hidden layers with 50 nodes each. Output with 7 nodes.</p>
<pre><code class="lang-auto"> self.hidden_layer_1 = torch.nn.Linear(78,50)
 self.hidden_layer_2 = torch.nn.Linear(50,50)
 self.output = torch.nn.Linear(50, 7)
</code></pre>
<pre><code class="lang-auto">def forward(self, input):
        weights = np.load('./weights.npy')
        weight = torch.from_numpy(weights)
        self.hidden_layer_1.weight = torch.nn.Parameter(weight)
        H1 = self.hidden_layer_1(input)
        H1 = self.ReLU(H1)
        H2 = self.hidden_layer_2(H1)
        H2 = self.ReLU(H2)
        final_inputs = self.output(H2)
        # not applying activation on final_inputs because CrossEntropyLoss does that
        return final_inputs
</code></pre>
<p>I’m initializing weights for the first layer and that’s giving me an error. The shape of the weights is (50,7).<br>
Error - return torch._C._nn.linear(input, weight, bias)<br>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1024x78 and 7x50)</p>
<p>Batch size is 1024, so that’s where the 1024 value comes from.</p>
<p>I am a bit confused what to do. What should the shape of my weights be so that I don’t get this error?<br>
I get these weights from running an unsupervised layer which is Restricted Boltzmann Machine.</p>

username: KFrank
cooked: <p>Hi jpj!</p>
<aside class="quote no-group quote-modified" data-username="jpj" data-post="1" data-topic="117636" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ed655f/48.png" class="avatar"> jpj:</div>
<blockquote>
<p>I have a fully connected neural network with 78 input features and two hidden layers with 50 nodes each. Output with 15 nodes.</p>
<pre><code class="lang-auto"> self.hidden_layer_1 = torch.nn.Linear(78,50)
 self.hidden_layer_2 = torch.nn.Linear(50,50)
 self.output = torch.nn.Linear(50, 15)
</code></pre>
</blockquote>
</aside>
<p>This is fine.  At this point <code>self.hidden_layer_1.weight</code> is a tensor with<br>
shape <code>[50, 78]</code>.  (The order of the dimensions is correct because of<br>
how pytorch multiplies batch tensors with weight tensors.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">def forward(self, input):
        weights = np.load('./weights.npy')
        weight = torch.from_numpy(weights)
        self.hidden_layer_1.weight = torch.nn.Parameter(weight)
</code></pre>
</blockquote>
</aside>
<p>This is your problem,  You are replacing <code>hidden_layer_1.weight</code>, whose<br>
shape is consistent with the number of features in your input batch, with<br>
a different tensor with a shape, <code>[50, 7]</code>, that is not consistent with your<br>
input batch.</p>
<aside class="quote no-group">
<blockquote>
<p>I’m initializing weights for the first layer and that’s giving me an error. The shape of the weights is (50,7).</p>
</blockquote>
</aside>
<p>Yes.  Presumably the file weights.npy contains a (numpy) tensor of<br>
shape <code>[50, 7]</code>.  The question is:  Is this what you want?  The numpy<br>
tensor would be appropriate if your input had 7 features (rather than 78).</p>
<aside class="quote no-group">
<blockquote>
<p>What should the shape of my weights be so that I don’t get this error?</p>
</blockquote>
</aside>
<p>If your input has 78 features, <code>hidden_layer_1.weight</code> should have<br>
shape <code>[50, 78]</code> – just as it does before you replace the weight tensor.</p>
<aside class="quote no-group">
<blockquote>
<p>I get these weights from running an unsupervised layer which is Restricted Boltzmann Machine.</p>
</blockquote>
</aside>
<p>Why the mismatch in the number of features?  Is your Restricted<br>
Boltzmann Machine processing data with 78 features or 7?</p>
<p>Best.</p>
<p>K. Frank</p>