Is it ok to pass the same parameter to optimizer multiple times?

hughperkins: <p>I’m thinking of writing an encoder-decoder, with shared embedding weights, like:</p>
<pre><code>embedding = nn.Embedding(input_size, hidden_size)
encoder = Encoder(embedding=embedding)
decoder = Decoder(embedding=embedding)
</code></pre>
<p>where eg Encoder <code>__init__</code> would do:</p>
<pre><code>def __init__(self, embedding):
    hidden_size = embedding.weight.size()[1]
    self.embedding = embedding
    self.enc_rnn = nn.RNN(hidden_size, hidden_size)
</code></pre>
<p>and then create the optimzer like:</p>
<pre><code>opt = optim.Adam(
    list(embedding.parameters()) +
    list(encoder.parameters()) +
    list(decoder.parameters())
)
</code></pre>
<p>To what extent is having the embedding parameters passed to the optimizer multiple times ok/ not ok? Any better way to handle this, in idiomatic pytorch? I want to avoid allocating the embedding multiple times, then deallocating the ones we dont need, ideally.</p>

hughperkins: <p>Ok, I have done this like this for now, unless anyone has a better idea?</p>
<pre><code>p = (
    set(encoder.parameters()) |
    set(decoder.parameters()) |
    set(embedding.parameters()))
opt = optimizer_fn(p, lr=0.001)</code></pre>

samarth-robo: <ol>
<li>If I’m not missing something obvious, do you really need to explicitly pass the <code>embedding.parameters()</code> to the <code>opt</code>, since you already gave it <code>encoder.parameters()</code> which includes <code>embedding.parameters()</code>?</li>
<li>The code for <code>nn.Module.__setattr__()</code> suggests that it will remove duplicate parameters passed to it, so it seems to me that</li>
</ol>
<pre><code class="lang-auto">opt = optim.Adam(list(encoder.parameters()) + list(decoder.parameters()))
</code></pre>
<p>should be fine.</p>

alexis-jacq: <aside class="quote no-group" data-username="samarth-robo" data-post="3" data-topic="4884">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samarth-robo/48/3487_2.png" class="avatar"> samarth-robo:</div>
<blockquote>
<p>The code for <code>nn.Module.__setattr__()</code> suggests that it will remove duplicate parameters passed to it, so it seems to me that</p>
<pre><code class="lang-auto">opt = optim.Adam(list(encoder.parameters()) + list(decoder.parameters()))
</code></pre>
</blockquote>
</aside>
<p>You can still have two modules with dictionaries as parameters. Both modules will have once each of its parameters in the dict, but if the two dictionaries have common parameters, it will raise error</p>
<pre><code class="lang-auto">ValueError: some parameters appear in more than one parameter group
</code></pre>

dgriff: <p>So how I would do that is set up model classes as such:</p>
<p>Class Encoder which has its own inits and then decoder class that has its own inits</p>
<p>Finally you can set up container module class EncDec with inits that you specify equal inits From Encoder and decoder class contained in EncDec class.</p>
<p>Model = ENcDec(“whatever arguments you need”)</p>
<p>Optimizer = optim.adam(Model, lr = 0.0001)</p>

karl7: <p>how to resolve that problem, i really need to optimize two groups of params…</p>