Trainable input z-vector to generator

sudocoder: <p>How do I get the input noise vector to a generator to train, while freezing the generator weights? I’ve been trying to set <code>requires_grad=True</code> for the input, freezing the model weights, and training. However, the input as I print it out does not change over the coarse of training (nor does the model), so I’m clearly missing something.</p>
<p>Do I have to use the input as a parameter into the optimizer when I initialize the optimizer (like in neural style transfer), and if so, how do I do that with MSELoss? Or is there a simpler method?</p>
<pre><code class="lang-auto"># Get the noise vector
z = util.get_input_noise() # vector of size 100 with values [0,1)
z = z.detach().requires_grad_()

# Freeze the model
for param in model.parameters():
    param.requires_grad = False

model.eval() # Can I use model in eval mode for this or does this do something weird with backprop?

# Training loop (simplified -- assume certain vars already initialized)
for i in range(n_iters):
    probs = model.forward(z)
    loss = torch.zeros(1, requires_grad=True).to(device)
    loss = loss_fn(probs, target)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<p>Thanks so much!</p>
<p>Related posts (did not work for me <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=14" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> ):</p><aside class="quote quote-modified" data-post="1" data-topic="2569">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yunjey/48/1386_2.png" class="avatar">
    <div class="quote-title__text-content">
      <a href="https://discuss.pytorch.org/t/how-can-i-train-input-not-weight/2569">How can i train input (not weight)?</a> 
    </div>
  </div>
  <blockquote>
    In GAN, i want to find latent vector z corresponding to the real image. One way to do this is to train z to minimize the error between sampled image and real image. 
However, when i ran the code below, Error message appeared: “ValueError: can’t optimize a non-leaf Variable”. 
targets             # target images of shape (batch_size, 3, 64, 64)

z = Variable(torch.randn(batch_size, 100), requires_grad=True).cuda()
optim = torch.optim.Adam([z], lr=0.01)            # This causes an error. (if i del…
  </blockquote>
</aside>
<aside class="quote" data-post="1" data-topic="15642">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/backpropper/48/77927_2.png" class="avatar">
    <div class="quote-title__text-content">
      <a href="https://discuss.pytorch.org/t/trainable-parameter-as-input/15642">Trainable parameter as input</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
    </div>
  </div>
  <blockquote>
    I can’t understand the difference: 
prior = Variable(torch.zeros(24), requires_grad=True)
prior_param = [nn.Parameter(prior.data)]
h_test = torch.autograd.grad(outputs=prior.mean(), inputs=prior_param) #does not work
h_test = torch.autograd.grad(outputs=prior.mean(), inputs=prior) #works

I want to make a trainable parameter as an input to the computation graph. 
The error is 
RuntimeError: One of the differentiated Variables appears to not have been used in the graph
  </blockquote>
</aside>

<p>Neural Style transfer: <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html#sphx-glr-advanced-neural-style-tutorial-py" class="inline-onebox" rel="noopener nofollow ugc">Neural Transfer Using PyTorch — PyTorch Tutorials 2.8.0+cu128 documentation</a></p>

JuanFMontesinos: <p>I’d say you have to parse the input to the optimizer. Are you doing so?</p>

sudocoder: <p>Yes, that’s what I was doing incorrectly - the optimizer needs to take in the input.</p>