How does optmizer work on sample specific parameters?

Sepehr_Amini_Afshar: <p>I am solving a computer vision task and here is the context:</p>
<ul>
<li>Assume we have a neural network where all the images will be passed through it during the training.</li>
<li>For image <em>i</em> there are several (0 to maybe 4) parameters which correspond to this single image. In other words, during training I want these parameters to be optimized as well.</li>
<li>The loss function for each image is the average of several constraints that apply to that image. Suppose we have a pool of ~10 constraints, and an image may be evaluated by any subset of these (this subset is <strong>not random</strong>—it is determined according to the labels available for the image). The average represents the loss value.</li>
<li>When I solve the problem mathematically, it is shown that the system is <strong>over-determined / over-constrained</strong>. Meaning, in theory, there should exist an optimal solution to minimize the loss function.</li>
</ul>
<hr>
<h3><a name="p-474524-my-questions-1" class="anchor" href="#p-474524-my-questions-1"></a>My questions:</h3>
<ol>
<li>
<p>Is the following a logical and correct way to use the optimizer?</p>
<pre data-code-wrap="python"><code class="lang-python">optimizer = Adam(parameters=list(model.parameters()) + image_specific_parameters_list, lr=...)
</code></pre>
</li>
<li>
<p>If I evaluate each image according to the loss function (explained above), does the optimizer—upon calling <code>backward()</code>—understand that it should only optimize the parameters that correspond to the batch images?</p>
<blockquote>
<p>Note: the model will be updated for all images, but those <strong>image-specific parameters</strong> must be updated only when the model sees those images during training.</p>
</blockquote>
</li>
</ol>

KFrank: <p>Hi Sepehr!</p>
<aside class="quote no-group" data-username="Sepehr_Amini_Afshar" data-post="1" data-topic="222538">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sepehr_amini_afshar/48/78015_2.png" class="avatar"> Sepehr_Amini_Afshar:</div>
<blockquote>
<ol>
<li>
<p>Is the following a logical and correct way to use the optimizer?</p>
<pre data-code-wrap="python"><code class="lang-python">optimizer = Adam(parameters=list(model.parameters()) + image_specific_parameters_list, lr=...)
</code></pre>
</li>
<li>
<p>If I evaluate each image according to the loss function (explained above), does the optimizer—upon calling <code>backward()</code>—understand that it should only optimize the parameters that correspond to the batch images?</p>
</li>
</ol>
</blockquote>
</aside>
<p>Yes – or at least maybe.</p>
<p>If you do not use momentum nor weight decay, then then calling <code>optimizer.step()</code> will</p>
<p>only modify parameters that have a non-zero <code>.grad</code>.  Note that <code>Adam</code> <em>does</em> use momentum,</p>
<p>so this particular optimizer probably won’t work for you.  I would use (or at least start with)</p>
<p><code>SGD</code> (without momentum nor weight decay).</p>
<p>If indeed your computed loss value only depends on an “image-specific parameter” when</p>
<p>that particular image contributes to that loss, then calling <code>loss.backward()</code> will only</p>
<p>populate such parameters – those whose images contribute to the loss – with a non-zero</p>
<p><code>.grad</code>.</p>
<p>So, yes, if done with these points in mind, your scheme should work.  Note that you can easily</p>
<p>test this.  Perform a backward pass and optimizer step and check that only the appropriate</p>
<p>image-specific parameters have in fact been modified.</p>
<p>Best.</p>
<p>K. Frank</p>