Optimization of inputs

username: Amiro
cooked: <p>Hi,</p>
<p>I have a Softmax model, can I calculate the gradients with respect to the input vectors so that I optimize the input vectors and the total loss?</p>
<p>through these steps, the loss is calculated (cross entropy) and the weights and biases are updated</p>
<p>loss = self.criterion(logits, labels) + self.regularizer<br>
loss.backward(retain_graph=True)<br>
self.optimizer.step()</p>
<p>How can I include input vectors in the optimisation process so that the model learns and updates: weights, biases, and input vectors?</p>
<p>Thanks</p>
<p>A</p>

username: Latope2-150
cooked: <p>Yes you certainly can.<br>
You simply need to set <code>requires_grad=True</code> for the input vectors as such:</p>
<pre><code class="lang-python">inputs.requires_grad_(True)
logits = model(inputs)
loss = self.criterion(logits, labels) + self.regularizer
loss.backward()
self.optimizer.step()
</code></pre>
<p>This will compute the gradient of the loss w.r.t. the inputs when you call <code>loss.backward()</code> (which will be stored in <code>inputs.grad</code>). You also need to add the inputs to your optimizer, otherwise, <code>optimizer.step()</code> will not modify them.</p>

username: Amiro
cooked: <aside class="quote no-group" data-username="Latope2-150" data-post="2" data-topic="70015">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/ed8c4c/48.png" class="avatar"> Latope2-150:</div>
<blockquote>
<p>add the inputs to your optimizer</p>
</blockquote>
</aside>
<p>Thank you so much for your reply, I already come to the same idea yesterday but I want to make sure of the following:</p>
<p>1- How can I add the input variable to the optimizer? I tried to use nn.parameter at initialization but clearly it seems that I can not initialize a variable that is not defined yet. Any help?</p>
<hr>
<p>is it simply like this?</p>
<p>data = Variable(sample[0], requires_grad=True)<br>
self.optimizer.add_param_group({“params”: data})</p>
<p>If yes, when I do: print (list(self.parameters())), I dont see the input among the parameters. Only weight and biases!</p>
<p>*** Also when i run the program with these 2 lines above it is taking very much long time. When adding the input of the model to the optimisation process, do i occupy the memory in any uncessary manner or is this slow running normal)?</p>
<p>Thanks for clarification</p>
<hr>
<p>2- I would like to confirm: adding the input to the optimizer will: Make the total loss optimized considering the input data, AND will update the input data as well (so that input data as well well not be the same)?</p>
<p>Thank you sooo much for your help</p>
<p>Am</p>

username: alwynmathew
cooked: <ol>
<li>Add <code>input</code> tensor to the optimizer:</li>
</ol>
<pre><code class="lang-auto">input.requires_grad = True
optimizer = optim.Adam([input], learning_rate)
</code></pre>
<p>Note that <code>Variables</code> don’t exist anymore in the newest versions on PyTorch.</p>
<p>In case, if you want to add <code>input</code> tensor along with <code>model</code> parameters to optimizer:</p>
<pre><code class="lang-auto">params_to_train = [input]
params_to_train += list(model.parameters())
optimizer = optim.Adam(params_to_train, learning_rate)
</code></pre>
<ol start="2">
<li>Yes,  adding the <code>input</code> to the optimizer will make <code>input</code> trainable. Make sure that <code>grad</code> of <code>input</code> tensor is turned on.</li>
</ol>
<p>This may help you:</p><aside class="quote quote-modified" data-post="1" data-topic="68722">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alwynmathew/48/8684_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/add-one-tensor-from-dataloader-to-optimiser/68722">Add one tensor from dataloader to optimiser</a> 
  </div>
  <blockquote>
    I want to optimize my init_tensor instead of optimizing the model. My data loader train_dataloader gives the initial value of init_tensor. 
model.eval()
for inputs in train_dataloader:
     init_tensor = inputs["init_tensor"]
     optimizer = optim.SGD(init_tensor, learning_rate)
     output = model(init_tensor)
     loss = loss_fn(output)
     model.zero_grad()
     loss.backward()
     optimizer.step()



Defining the optimizer inside the training loop doesn’t look right. Will this hinder opti…
  </blockquote>
</aside>


username: Latope2-150
cooked: <p>The difficulty in helping you here is that you are not giving a fully working code sample so I am not sure what you exactly wish to do.<br>
I would also strongly suggest that you understand the way the optimizer are implemented in PyTorch.</p>
<p>In your case, if the input is not changing (not using a dalaloader for example as you would load new data at each iteration) ; you’d need to add the inputs to the optimizer when you are defining it:</p>
<pre><code class="lang-python">model = MyModelClass() # initialize model
inputs = MyData() # load data
inputs.requires_grad_(True)

parameters = [
    {'params': model.parameters()},
    {'params': [inputs], 'lr': inputs_learning_rate}
]
optimizer = SGD(parameters, lr=learning_rate)
</code></pre>
<p>In this case, the learning rate for the parameters of the model pour be <code>learning_rate</code> while the learning rate for the inputs would be <code>inputs_learning_rate</code>.</p>
<p>The reason you are not seeing the inputs among the parameters when calling <code>list(self.parameters())</code> is probably because you are in the model class which has parameters as attributes but your data is not an attribute of the class. If you want a list of the parameters that are being optimized by the optimizer, you need to call <code>optimizer.param_groups</code>.</p>

username: Amiro
cooked: <p>Here is a part of my code</p>
<p>** the Optimizer is defined outside this block of code **</p>
<hr>
<p>for epoch in range(num_epochs):</p>
<pre><code>         print("epoch no. ", epoch + 1)    # training loop
         train_loss = 0
         self.model.train()
         New_data=[]
         for batch_ndx, sample in enumerate(TrainLoader):
            
             self.optimizer.zero_grad()
             data = Variable(sample[0], requires_grad=True)
             self.data = nn.Parameter(data)
             self.optimizer.add_param_group({"params": data})
             labels = sample[1].type(torch.LongTensor)-1
             logits, probas = self.model(data.float())
             loss = self.criterion(logits, labels) + self.regularizer
             loss.backward(retain_graph=True)
             self.optimizer.step()
             if epoch + 1 == num_epochs:
                 data_optimised = list(self.parameters())[0].clone()
                 New_data.append(data_optimised.detach().numpy())     
             train_loss += loss.item()
</code></pre>
<p>etc</p>
<hr>
<p>The purpose of “New_data” is to return the optimized input data so that I can compare to the original data before optimization. There are 2 for loops; one over epochs and one over data batches, and I only append the optimised data at the last epochs of training.  ***** Does this seem logical?? *****</p>
<p>What i observed is that the difference between input data and the optimised data is so small and it did not affect the result of logistic regression remarkably. Consequently, I want to make sure:</p>
<p>1- If my implementation is correct in the way of how I add data to the optimizer and optimise them?<br>
2- If data optimization can not so much affect the results of regression, what is the utility of this step in general? I remarked some people trying to do the same but not a lot.</p>
<p>********** Another important question **************<br>
In the code above it was easy to add (requires_grad=True) to the data tensor. This data is produced through cross correlating word vectors (i.e., word_vectors_1 * word_vector_2 = data) . This process of cross correlation is coded outside the training phase. Can I include word_vectors_1 &amp; word_vector_2 in the optimization process as well? I wont be able to add them to the optimizer as they are not included in the training file, where is the optimizer is first defined.</p>
<p>So how can I add them to the optimizer while they are defined in other files in the program?</p>
<p>Thanks for help</p>
<hr>
<p>Thanks</p>

username: kHan
cooked: <p>I have a similar problem.<br>
To get the gradient w.r.t input, I add the .requires_grad<br>
but it doesn’t work.</p>
<p>for num,(sample_img, sample_label) in enumerate(mnist_test):<br>
if num == 1:<br>
break</p>
<pre><code>sample_img = sample_img.to(device)
sample_img.requires_grad = True
prediction = model(sample_img.unsqueeze(dim=0))
cost = criterion(prediction, torch.tensor([sample_label]).to(device))
optimizer.zero_grad()
cost.backward()
print(sample_label)
print(sample_img.shape)

plt.imshow(sample_img.detach().cpu().squeeze(), cmap='gray')
plt.show()</code></pre>