Issue with sharing block - getting warning - issue 40967

philiptsmith: <p>Forum,</p>
<p>I am running the code from the VAE-GAN from the INTERSPEECH 2020 paper <a href="http://www.interspeech2020.org/uploadfile/pdf/Thu-3-4-11.pdf" rel="noopener nofollow ugc">Voice Conversion Using Speech-to-Speech Neuro-Style Transfer</a>.   This also happens with the UNIT paper code.</p>
<p>Pytorch is giving a warning as follows:<br>
UserWarning: optimizer contains a parameter group with duplicate parameters; in future, this will cause an error; see <a href="http://github.com/pytorch/pytorch/issues/40967" class="inline-onebox" rel="noopener nofollow ugc">Inconsistent behaviour when parameter appears multiple times in parameter list · Issue #40967 · pytorch/pytorch · GitHub</a> for more information</p>
<p>The problem is with the shared block and it appears that this is going to be error moving forward yet I’m unsure how to fix the issue so I van continue using pytorch for these types of structures.  If it is going to error out I will need an understanding how to get this to work.</p>
<p>The block is shared:</p>
<p>shared_dim = opt.dim * 2 ** opt.n_downsample</p>
<h1>
<a name="initialize-generator-and-discriminator-1" class="anchor" href="#initialize-generator-and-discriminator-1"></a>Initialize generator and discriminator</h1>
<p>encoder = Encoder(dim=opt.dim, in_channels=opt.channels, n_downsample=opt.n_downsample)</p>
<p>shared_G = ResidualBlock(features=shared_dim)</p>
<p>G1 = Generator(dim=opt.dim, out_channels=opt.channels, n_upsample=opt.n_downsample, shared_block=shared_G)</p>
<p>G2 = Generator(dim=opt.dim, out_channels=opt.channels, n_upsample=opt.n_downsample, shared_block=shared_G)</p>
<p>D1 = Discriminator(input_shape)</p>
<p>D2 = Discriminator(input_shape)</p>
<p>The warning is occurring when the Adam optimizer is being defined:</p>
<h1>
<a name="optimizers-2" class="anchor" href="#optimizers-2"></a>Optimizers</h1>
<p>optimizer_G = torch.optim.Adam(<br>
<strong>itertools.chain(encoder.parameters(), G1.parameters(), G2.parameters()),</strong><br>
lr=opt.lr,<br>
optimizer_G, lr_lambda=LambdaLR(opt.n_epochs, opt.epoch, opt.decay_epoch).step<br>
betas=(opt.b1, opt.b2),<br>
)<br>
optimizer_D1 = torch.optim.Adam(D1.parameters(), lr=opt.lr, betas=(opt.b1, opt.b2))<br>
optimizer_D2 = torch.optim.Adam(D2.parameters(), lr=opt.lr, betas=(opt.b1, opt.b2))</p>
<h1>
<a name="learning-rate-update-schedulers-3" class="anchor" href="#learning-rate-update-schedulers-3"></a>Learning rate update schedulers</h1>
<p>lr_scheduler_G = torch.optim.lr_scheduler.LambdaLR(<br>
optimizer_G, lr_lambda=LambdaLR(opt.n_epochs, opt.epoch, opt.decay_epoch).step</p>
<p>The G1 parameters and G2.parameters have a shared block of weights and this I believe is causing the issue.  Duplicate parameters? I haven’t been able to find any remedy to this issue, yet the sharing of blocks does occur in many of the newer papers.</p>
<p>Any suggestions on how to get around this warning?</p>
<p>Bests,<br>
Philip</p>

tom: <p>So coceptually, what do you want to happen if the ‘optimal’ configuration for g1 and g2 happens to be at different values of the shared block?</p>
<p>If you duplicate, you will get different values.</p>
<p>If you just want the shared block to be pulled in the respective descent direction with each step, without caring too much about the details, having two optimizers might be better.</p>
<p>If you care about one more than the other, you might use a weighted loss of those for g1and g2 and filter out the duplicates.</p>
<p>The last two are not really excluding each other: you could split out the common part to a third parameter group.</p>
<p>Best regards</p>
<p>Thomas</p>