How to extend a tensor

username: FA_mn
cooked: <p>0</p>
<p>I want to extend a tensor in PyTorch in the following way:</p>
<p>Let <code>C</code> be a <code>3x4</code> tensor which <code>requires_grad = True</code>. I want to have a new <code>C</code> which is <code>3x5</code> tensor and <code>C = [C, ones(3,1)]</code> (the last column is a one-vector, and others are the old <code>C</code>) Moreover, I need <code>requires_grad = True</code> for new <code>C</code>.</p>
<p>I need to use this tensor as a parameter.</p>
<p>In a first step, I optimize the parameters in <code>N</code> epochs.<br>
In the second step, I need to extend the parameters (something I sad before, and I used<br>
<code>torch.nn.functional.pad</code> and the parameters are extended)<br>
In the third step, I should optimize the extended parameters, but parameters are the same as before and not extended.</p>
<p>Is it possible to inform me about this problem?</p>

username: caonv
cooked: <p>Did you try to set <code>requires_grad=True</code> for the extended tensor?<br>
Another possible way is that you extend <code>C.data</code> instead of <code>C</code> directly.</p>

username: FA_mn
cooked: <p>Thanks for your reply.<br>
I used :</p>
<pre><code class="lang-auto">param = torch.nn.functional.pad(param, (0,1,0,0), "constant", 1.0).detach().requires_grad_()
</code></pre>
<p>Then check the shape and it is correct.<br>
but after the optimization the parameters are the same as before and not extended.</p>

username: caonv
cooked: <p>Don’t detach the tensor.</p>

username: FA_mn
cooked: <p>I appreciate your time to help me.<br>
even I do not detach it, it does not work and the parameters are same size as before extending.</p>
<p>could you please have a look at part of my code:</p>
<pre><code class="lang-auto">   for epoch in range(1, n_epochs+1):
        
        
        train_loss = calc_forward(train_tu, train_tc, model, loss_fn, is_train=True)
        val_loss = calc_forward(val_tu, val_tc, model, loss_fn, is_train=False)
        
        
        optimizer.zero_grad()
        train_loss.backward()
        optimizer.step()
        
        
     
    for name, param in model.named_parameters():

        if name == '0.weight':
            
            param = torch.nn.functional.pad(param, (0,0,0,1), "constant", 1.0).requires_grad_()
            
        if name == '0.bias':
            
            param = torch.nn.functional.pad(param, (0,1), "constant", 1.0).requires_grad_()

        if name == '2.weight':
            
            param = torch.nn.functional.pad(param, (0,1,0,0), "constant", 1.0).requires_grad_()

        if name == '2.bias':
            print(name, param.shape, param)
            param = torch.nn.functional.pad(param, (0,1), "constant", 1.0).requires_grad_()
            print(param.shape, param)
            
    for epoch in range(1, n_epochs+1):
        
        
        train_loss = calc_forward(train_tu, train_tc, model, loss_fn, is_train=True)
        val_loss = calc_forward(val_tu, val_tc, model, loss_fn, is_train=False)
        
        
        optimizer.zero_grad()
        train_loss.backward()
        optimizer.step()
        
        
    for name, param in model.named_parameters():
        print(name, param.shape, param)
</code></pre>
<p>my model is really simple:</p>
<pre><code class="lang-auto">model = nn.Sequential(nn.Linear(1, 4), nn.Tanh(), nn.Linear(4, 1))
</code></pre>

username: caonv
cooked: <p>I don’t think your way of modifying the weight is correct. You should try <code>model.layer.weight = [.....]</code> and <code>model.layer.bias = [....]</code></p>

username: FA_mn
cooked: <p>Yes, you’re right.<br>
I should assign the parameters value, not the <code>param</code>. Thanks a lot for informing me.<br>
I need to find a way to assign the parameters of the model.</p>