What is the proper ways to set parameters?

The_duke: <p>Hello all,</p>
<p>Happy new year,</p>
<p>I hope you all had a grand time. 2022 greeted me with a problem:<br>
this is my neural network.</p>
<p>class NeuralNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.flatten = nn.Flatten()<br>
self.linear_relu_stack = nn.Sequential(<br>
nn.Linear(5, 8),<br>
nn.ReLU(),<br>
nn.Linear(8, 12),<br>
nn.ReLU(),<br>
nn.Linear(12, 2),<br>
nn.ReLU()<br>
)</p>
<pre><code>def forward(self, x):
    x = self.flatten(x)
    logits = self.linear_relu_stack(x)
    return logits
</code></pre>
<p>I wrote this function to change its weights.<br>
In a nutshell: it adds up the different parameter tensors, flattens them, modify them a bit and put them back together in the model.</p>
<p>def jiggle(x, y, z):<br>
<span class="hashtag">#E_1</span>, E_2, E_3 are orthogonal vectors in R^3 / 3D<br>
x_coord = (torch.tensor(E_1) * torch.tensor(x))<br>
y_coord = torch.tensor(E_2) * torch.tensor(y)<br>
z_coord = torch.tensor(E_2) * torch.tensor(z)</p>
<pre><code>x_coord = x_coord.to(device)
y_coord = y_coord.to(device
z_coord = z_coord.to(device)

flat_shifted_params = torch.cat(flat_params) * x_coord + torch.cat(flat_params) + torch.cat(flat_params) * z_coord
split_params = torch.split(flat_shifted_params, elem_count)
reshaped_params = [tensor.reshape(shapes[idx]) for idx, tensor in enumerate(split_params)]

def set_weights(i, j, model):
    if i &lt; LEN_MODEL:
        if hasattr(model.linear_relu_stack[i], 'weight'):
            model.linear_relu_stack[i].weight = nn.Parameter(reshaped_params[j])
            j += 1
            i += 1
            set_weights(i, j, model))
        else:
            i += 1
            set_weights(i, j, model)

set_weights(0, 0, model)
</code></pre>
<p>When I try to test, I get this traceback:</p>
<p>Traceback (most recent call last):<br>
File “C:/Users/…/Desktop/Raytracer Pytorch parameter space visualization/main.py”, line 314, in <br>
score = test(test_dataloader, model, LOSS_FN)<br>
File “C:/Users/…/Desktop/Raytracer Pytorch parameter space visualization/main.py”, line 101, in test<br>
pred = model(X)<br>
File “D:\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “C:/Users/…/Desktop/Raytracer Pytorch parameter space visualization/main.py”, line 86, in forward<br>
logits = self.linear_relu_stack(x)<br>
File “D:\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “D:\anaconda3\lib\site-packages\torch\nn\modules\container.py”, line 139, in forward<br>
input = module(input)<br>
File “D:\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “D:\anaconda3\lib\site-packages\torch\nn\modules\linear.py”, line 96, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “D:\anaconda3\lib\site-packages\torch\nn\functional.py”, line 1847, in linear<br>
return torch._C._nn.linear(input, weight, bias)<br>
RuntimeError: mat2 must be a matrix, got 1-D tensor<br>
when I feed it to the testing function.</p>
<p>Python doesn’t complain if I comment out the jiggle() function I wrote, so clearly I did something wrong here.</p>
<p>I assume mat2 refer to the parameter weights. So I tried debugging and set a breakpoint right after     def set_weights(i, j, model):, the console command :</p>
<p>[x for x in model.parameters()] == [nn.Parameter(x) for x in reshaped_params]</p>
<p>returns True.</p>
<p>I’m not sure why is it complaining that it is different?</p>

ptrblck: <p>I think the parameter check is performed after you’ve flattened the parameters already, so while it would return <code>True</code>, I guess flattening the parameters in the first place creates the shape mismatch error. Check the shape of each parameter and make sure the linear layer weights are 2 dimensional.</p>

The_duke: <p>I inspected the layer one by one after .<br>
It turns out the solution was even simpler than this: I forgot to set the bias,  so I inadventantly  chucked a 1-D bias tensor into the slot for weight (where it of course expected a matrix).</p>
<p>Rewriting the function set_weights as such did the trick:</p>
<pre><code>def set_weights(i, j):
    if i &lt; LEN_MODEL:
        if hasattr(model.linear_relu_stack[i], 'weight'):
            model.linear_relu_stack[i].weight = nn.Parameter(reshaped_params[j])
            j += 1
            model.linear_relu_stack[i].bias = nn.Parameter(reshaped_params[j])
            i += 1
            j += 1
            set_weights(i, j)
        else:
            i += 1
            set_weights(i, j)
</code></pre>
<p>I just lost track of things due to sleep deprivation.</p>