Data Tensor Optimisation

Amiro: <p>Hi,</p>
<p>I used to optimize a data tensor after marking it with ( requires_grad= True) and adding it to the optimizer which is defined in the training loop in the same (file.py).</p>
<p>Now I would like to add to this optimization process another data tensor in another different (file.py), how can I add it to the same optimizer?</p>
<p>Thanks for your help</p>

ptrblck: <p>If you have defined the other tensor in e.g. a function from the other file, you could import this function and get the tensor in your current script.</p>
<p>Let me know, if I misunderstood your use case.</p>

Amiro: <p>Hi,</p>
<p>Thanks a lot for your information, I have I have tried to make it and included that tensor in the optimization process. In the part of the attached code, that data tensor that I wanted to include in the optimization process is the (data_words) similarly to (data_phrases).  Data_phrases is optimized successfully and data is updated, so that when I receive the output tensor ( New_phrases_optimized) data is clearly different. However, for (data_words) it is never optimized, so the input data is exactly the same as the output data. Is there any mistake here?</p>
<p>(reminder: the other part of the code is about softmax regression for the phrases, so that the purpose of having words here is only to include in the optimisation process). Should I add (data_words) to the optimizer  in a different way??</p>
<hr>
<p>for epoch in range(num_epochs):</p>
<pre><code>        print("epoch no. ", epoch + 1)    # training loop
        train_loss = 0
        self.model.train()
        New_words_optimized=[]
        New_phrases_optimized=[]
        for batch_ndx, sample in enumerate(zip(TrainLoader,WordsLoader)):
            self.optimizer.zero_grad()
            data_phrases = Variable(sample[0][0], requires_grad=True)
            data_words = Variable(sample[1][0], requires_grad=True)
            self.data_phrases  = nn.Parameter(data_phrases)
            self.data_words = nn.Parameter(data_words)
            self.optimizer.add_param_group({"params": data_phrases})
            self.optimizer.add_param_group({"params": data_words})
            labels_phrases = sample[0][1].type(torch.LongTensor)-1
            logits, probas = self.model(data_phrases .float())
            loss = self.criterion(logits, labels_phrases) + self.regularizer
            loss.backward(retain_graph=True)
            self.optimizer.step()

            if epoch + 1 == num_epochs:

               phrases_optimized = list(self.parameters())[0].clone()
               words_optimized = list(self.parameters())[1].clone()
               New_phrases_optimized.append(phrases_optimized.detach().numpy())
               New_words_optimized.append(words_optimized.detach().numpy())
            
            train_loss += loss.item()
</code></pre>
<hr>
<p>Thanks a lot</p>
<p>AA</p>

ptrblck: <p>Could you check the <code>.grad</code> attribute of <code>data_words</code> after the <code>backward</code> call?</p>
<p>Also, <code>Variables</code> are deprecated since PyTorch <code>0.4.0</code>, so you can use tensors directly now.<br>
The usual approach would be to pass the parameters as a list to the initialization of your optimizer, but your approach seems to work for <code>data_phrases</code>.</p>

Amiro: <p>Hi</p>
<p>I checked and the tensor of data_words gives “None”, so it was clear that it is not added to the optimizer appropriately. So I came to these 2 lines:</p>
<p>self.optimizer.add_param_group({“params”: data_phrases})<br>
self.optimizer.add_param_group({“params”: data_words})</p>
<p>and replaced (data_phrases) with (data_words) and when I checked the gradient of (data_words) it was showing values and not “None” while (data_phrases) was the one whose .grad gives “None” this time.</p>
<p>So I guess I need to pass the parameters to the optimizer in a different way? May be in one instruction together?</p>
<p>Thanks in advance for your help</p>
<p>Amir</p>

ptrblck: <p>Try to pass them as a list during initialization:</p>
<pre><code class="lang-python">optimizer = optim.SGD([data_phrases, data_words], lr=1.)
</code></pre>

Amiro: <p>Thank you, the problem is that the intialization step is</p>
<p>self.optimizer = O.optimizer(self.model.parameters(), lr, momentum, optimizer_name)</p>
<p>that i have before the training loop. If I use data_phrases and data_words before they are being defined, there might be an error.</p>
<p>Cant I do something like “self.optimizer.add_param_group({“params”: [data_phrases, data_words]})” in the training phase? I tried that but still not working!</p>
<p>Any advise?</p>
<p>Thanks a lot</p>