Fail to train nn.Parameter()

lucastononrodrigues: <p>Inside my nn.Module class I am creating nn.Parameters() such as:</p>
<blockquote>
<p>self.B =self.reset_parameters(nn.Parameter(torch.Tensor(self.projection_dim,10),requires_grad=True)).unsqueeze(0).repeat(num_heads,1,1) <span class="hashtag-raw">#I</span> am using the reset parameters from torch source code <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L81" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L81</a></p>
</blockquote>
<p>Apparently, when I run the training loop I get that error.<br>
My goal is to train two matrices A and B (like a feedforward neuralnet) in order to project a given x as x<em>A</em>B, it has to compute AB and then xAB for complexity reasons.</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “main.py”, line 304, in <br>
main(args)<br>
File “main.py”, line 291, in main<br>
training(model,criterion,optimizer,scheduler,train_loader,valid_loader,args)<br>
File “main.py”, line 221, in training<br>
loss.backward()<br>
File “C:\Users\lucas\anaconda3\envs\tcc\lib\site-packages\torch\tensor.py”, line 185, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “C:\Users\lucas\anaconda3\envs\tcc\lib\site-packages\torch\autograd_<em>init</em>_.py”, line 125, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.</p>
</blockquote>
<p>When I use retain_graph I get out of memory from cuda.<br>
The parameters self.A and self.B don’t show up in model.parameters() as well.</p>
<p>I’ve seen people training positional embedding like this and never had to retain_graph, so I wonder what could be wrong. Thanks</p>

ruotianluo: <aside class="quote no-group" data-username="lucastononrodrigues" data-post="1" data-topic="110295">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lucastononrodrigues/48/23856_2.png" class="avatar"> lucastononrodrigues:</div>
<blockquote>
<p>self.B =self.reset_parameters(nn.Parameter(torch.Tensor(self.projection_dim,10),requires_grad=True)).unsqueeze(0).repeat(num_heads,1,1)</p>
</blockquote>
</aside>
<p>Not sure what is the problem extractly, but try to do this:</p>
<p><code>self.B = nn.Parameter(torch.Tensor(self.projection_dim,10))</code><br>
(The way you use reset_parameters is wierd, I don’t even know why it works. You should just run self.reset_parameters at the end of <strong>init</strong>)</p>
<p>in the forward use,<br>
<code>self.B.unsqueeze(0).repeat(num_heads,1,1)</code></p>

lucastononrodrigues: <p>The way I reset parameters is the same from the nn.Linear</p>
<blockquote>
<p>def reset_parameters(self,W,B=None):<br>
W=torch.nn.init.kaiming_uniform_(W, a=math.sqrt(3))<br>
if B is not None:<br>
fan_in, _ = torch.nn.init.<em>calculate_fan_in_and_fan_out(W)<br>
bound = 1 / math.sqrt(fan_in)<br>
B=torch.nn.init.uniform</em>(B, -bound, bound)<br>
return W,B<br>
return W</p>
</blockquote>
<p>I will try without the reset_parameters and as soon as my gpus are free, thank you.</p>

ruotianluo: <p>What about not taking and returning anything, do exact what nn.Linear does.</p>

lucastononrodrigues: <p>In a certain way I am trying to use a projection just like nn.Linear and then projecting again with another matrix, but I would like to control the order of the computing, for time complexity reasons. x (W_1 W_2) instead of (x W_1) W_2</p>

ruotianluo: <p>I understand that. But reset_parameter should just initialize the parameter in place, instead of taking some input and return some output.</p>
<p>When you define a variable in nn.Module, nn.Module will do something secretly, so the best way to define a parameter is:<br>
<code>self.xxx = nn.Parameter(some tensor)</code><br>
instead of<br>
<code>self.xxx = nn.Parameter(soem tensor).unsqueeze(xxx).repeat(xxxx)</code></p>