Optimization of input tensors

username: Amiro
cooked: <p>Hi,</p>
<p>I wanted to add 2 data tensors to the optimizer (data_words &amp; data_phrases) but only (data_phrases) is optimized and not (data_words). Is there any mistake in the way I added them to the optimizer as in the sample code below (from inside the training loop of a softmax model)?</p>
<hr>
<p>self.optimizer.zero_grad()<br>
data_phrases = Variable(sample[0][0], requires_grad=True)<br>
data_words = Variable(sample[1][0], requires_grad=True)<br>
self.data_phrases  = nn.Parameter(data_phrases)<br>
self.data_words = nn.Parameter(data_words)<br>
self.optimizer.add_param_group({“params”: data_phrases})<br>
self.optimizer.add_param_group({“params”: data_words})<br>
labels_phrases = sample[0][1].type(torch.LongTensor)-1<br>
logits, probas = self.model(data_phrases .float())<br>
loss = self.criterion(logits, labels_phrases) + self.regularizer<br>
loss.backward(retain_graph=True)<br>
self.optimizer.step()</p>
<hr>
<p>Best</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>data_words</code> is not used to compute the loss in your example. So that would explain why it does not get any gradients and is not updated.</p>

username: Amiro
cooked: <p>Hi Alban,</p>
<p>thanks for your email,</p>
<p>I see your point but I guess we can simply calculate the gradient of  any tensor right? does not need to be involved directly in calculating the loss right?</p>
<p>Trying to check this point further, I tried to see the (.grad) of the tensors (data_phrases) and (data_words) and found that the gradients of the tensor (data_phrases) are updated unlike those of the tensor (data_words) it is always = None.</p>
<p>But coming to these 2 lines:</p>
<p>self.optimizer.add_param_group({“params”: data_phrases})<br>
self.optimizer.add_param_group({“params”: data_words})</p>
<p>I guess the matter is related to a problem in passing the tensor (data_words) to the optimizer.  For some reason only the (tensor_phrases) is passed correctly to the optimizer, but not the tensor (data_words). to check further, instead of trying to pass the tensors (data_phrases &amp; data_words) one after one, I only passed one tensor (data_words) alone, and in this case it worked well and I saw gradients were calculated.</p>
<p>So my question turns to be: How can I pass multiple tensors together to the optimizer? I tried many times but it does not work.</p>
<p>Any advise?</p>
<p>Thanks</p>
<p>A</p>

username: albanD
cooked: <blockquote>
<p>I see your point but I guess we can simply calculate the gradient of any tensor right? does not need to be involved directly in calculating the loss right?</p>
</blockquote>
<p>Well if the Tensor is not involved in the loss computation, it’s gradient will be 0s everywhere (or None, it is the same thing).<br>
So to have meaningful gradients, you need to use the Tensor, otherwise, they will always be 0 and you won’t be able to learn anything.</p>

username: Amiro
cooked: <p>Hi Alban</p>
<p>Actually I have calculated the (data_phrases) from (data_words) but this was outside the training loop.  So what I was intending to do is to pass to the optimizer the same batch of (data_words) that produced (data_phrases). So it is indirect relationship.  so that the optimizer updates the “data_phrases” (directly used to calculate the loss) and “data_words” (indirectly related).</p>
<p>The point I passed the tensor (data_words) to the training loop while not being concerned with the softmax model is that I needed to pass it to the same optimiser, therefore I made it this way.</p>
<p>still no sense?</p>
<p>I see your point, but then why if i only pass to the optimizer the tensor (data_words) self.optimizer.add_param_group({“params”: data_words}), I find that the gradients are calculated successfully (and not None)?</p>

username: albanD
cooked: <aside class="quote no-group" data-username="Amiro" data-post="5" data-topic="70938">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/cc9497/48.png" class="avatar"> Amiro:</div>
<blockquote>
<p>I see your point, but then why if i only pass to the optimizer the tensor (data_words) self.optimizer.add_param_group({“params”: data_words}), I find that the gradients are calculated successfully (and not None)?</p>
</blockquote>
</aside>
<p>Most likely because you also remove other code like the wrapping in the <code>nn.Parameter()</code> for <code>data_phrases</code> that disconnects data_phrases from data_words.</p>
<blockquote>
<p>Actually I have calculated the (data_phrases) from (data_words) but this was outside the training loop.</p>
</blockquote>
<p>If <code>data_phrases</code> is computed from <code>data_words</code>, why should it be in the optimizer? Shouldn’t you just update <code>data_words</code> and recompute <code>data_phrases</code> every time <code>data_words</code> is updated?</p>

username: Amiro
cooked: <p>I could not understand this point (Most likely because you also remove other code like the wrapping in the  <code>nn.Parameter()</code>  for  <code>data_phrases</code>  that disconnects data_phrases from data_words.)  I dont remove any thing indeed, exactly same code but instead of passing:</p>
<p>self.optimizer.add_param_group({“params”: data_phrases})<br>
self.optimizer.add_param_group({“params”: data_words})</p>
<p>I pass</p>
<p>self.optimizer.add_param_group({“params”: data_words})</p>
<p>What is remarkable is that If i change the order with respect to the first case:</p>
<p>self.optimizer.add_param_group({“params”: data_words})<br>
self.optimizer.add_param_group({“params”: data_phrases})</p>
<p>only (data_words) will be updated and (data_phrases) will give None, unlike the first case, which made me doubt about these 2 lines and how they are passed to the optimizer.  I feel confused!</p>
<p>actually it is possible to ignore (data_phrases) and not pass it to the optimiser as you mentioned. but the point is that it is a softmax process for phrases (not for words), i.e., I can try to update only the tensor (data_words) and use the updated words to produce phrases. BUT, the loss function in this case will have as input the tensor (data_phrases) and not (data_words), which takes us back to your argument about optimizing a tensor not involved in the calculation of loss.</p>
<p>What do you think?</p>

username: albanD
cooked: <p>Ok,<br>
This is weird.<br>
Could you cleanup your code to do the following:</p>
<ul>
<li>remove <code>Variable</code> as it is not needed anymore.</li>
<li>set <code>requires_grad=True</code> <em>only</em> for Tensor for which your want the <code>.grad</code> field to be populated.</li>
<li>Do not wrap things in nn.Parameter. This is onlyu needed if you are in an nn.Module and plan on calling <code>mod.parameters()</code> to get all the parameters.</li>
</ul>

username: ptrblck
cooked: <p>For completeness: also partially answered <a href="https://discuss.pytorch.org/t/data-tensor-optimisation/70491/7">here</a>.</p>

username: Amiro
cooked: <p>Hi,</p>
<p>I have done all this but still not able to resolve the point! the (.grad) of words are still not calculated. So I need to pass the tensors of phrases and words together to the optimizer.</p>
<p>Any help?</p>
<p>Regards</p>
<p>A</p>