Update the input of model(output of training model as input to another pretrained model) in a loop

KURUVILLA_ABRAHAM: <pre><code>    for batch_idx,img in enumerate(loaders['rgbimage_dataloader']):
        optimizer.zero_grad()
        # move to GPU
        if use_cuda:
            real = img[0].to(device)
            sketch_image = cycle_model(real)
        else:
            real = img[0].to(device)
            sketch_image = cycle_model(real)
        
        styleGan_output = model(real,sketch_image,Target_color.float(),noise)
       ####Pretrained model#################
        fakeEdge_image = cycle_model(styleGan_output)   
        
        Edge_loss = criterion(fakeEdge_image,sketch_image)   
        total_loss = Edge_loss
        total_loss.backward()
        optimizer.step()
        
       ####This is the way i am saving ########################
       torch.save(model.state_dict(),f"model_history/mu_stylegan{epoch}.pth")
</code></pre>
<p>In this way is the the model saving state of both the pretrained model as well ?<br>
or How could we do that ?</p>

ptrblck: <p>I don’t know which model is the pretrained one, but in your current code you are storing the <code>state_dict</code> of <code>model</code> only, which would contain all properly registered parameters, buffers, and other submodules.</p>

KURUVILLA_ABRAHAM: <p>Hi Patrick,</p>
<p>The cycle model is the pretrained one in the script .</p>
<p>Is it necessary to save this ?</p>

KURUVILLA_ABRAHAM: <p>Problem is that if I am using criterion as above my model is not improving.</p>
<p>While if am using below (not their in above script)</p>
<p>Criterion(styleganoutput,real) it improves .<br>
(I don’t want to use the above Criterion as I need to disentangle the output)</p>

ptrblck: <p>If <code>cycle_model</code> has parameters or buffers you would also need to save its <code>state_dict</code>.<br>
Make sure that all parameters get a valid gradient after the first <code>backward</code> call and thus that the computation graph is not detached.</p>

KURUVILLA_ABRAHAM: <p>Ok patrick…</p>
<p>I have few transformations done after getting the model output and cycle gan like getting color information and then using it as a criterion.</p>
<p>Is this state also save ,if not then how is the process of saving ?</p>

ptrblck: <p>I’m not exactly sure which issue you are currently trying to solve.</p>
<p>To save and restore the training you would need to save all trained parameters, buffers, etc.<br>
To do so, you would need to save the <code>state_dict</code> from all models, the optimizer, and schedulers.<br>
This will make sure that the internal “states” of these objects are properly restored and you could continue the training or perform the inference.</p>
<p>On the other hand, if you are trying to debug the issue that your models are not training properly, make sure the computation graph is not detached e.g. by checking the <code>.grad</code> attributes of all parameters which should be trained.</p>
<aside class="quote no-group" data-username="KURUVILLA_ABRAHAM" data-post="6" data-topic="142514">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kuruvilla_abraham/48/45695_2.png" class="avatar"> KURUVILLA_ABRAHAM:</div>
<blockquote>
<p>I have few transformations done after getting the model output and cycle gan like getting color information and then using it as a criterion.</p>
</blockquote>
</aside>
<p>If these transformations do not have an internal state you wouldn’t have to save anything (there won’t be anything to save besides the actual source code).</p>

KURUVILLA_ABRAHAM: <p>Hi Patrick,</p>
<p>Apologies for the confusion this is my entire training loop.</p>
<p>##<span class="hashtag">#Below</span> if my Code</p>
<pre><code>   valid_loss_min = valid_loss_min_input 

for epoch in range(start_epochs, n_epochs+1):
    # initialize variables to monitor training and validation loss
    train_loss = 0.0
     ###################
    # train the model #
    ###################
    model.train()
   
    for batch_idx,img in enumerate(loaders['rgbimage_dataloader']):
        optimizer.zero_grad()
        # move to GPU
        if use_cuda:
            real = img[0].to(device)
            sketch_image = cycle_model(real)
        else:
            real = img[0].to(device)
            sketch_image = cycle_model(real)
       
       ###Getting output from main model#####
        styleGan_output = model(real,sketch_image,Target_color.float(),noise)
     ###Feeding to the pretrained model########
        fakeEdge_image = cycle_model(styleGan_output)            
</code></pre>
<h1>
<a name="unormalizing-image-to-feed-to-pil-image-1" class="anchor" href="#unormalizing-image-to-feed-to-pil-image-1"></a>##<span class="hashtag">#Unormalizing</span> Image to feed to PIL image</h1>
<pre><code>        x_tensor = transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])(styleGan_output)
        min_i = x_tensor.min(dim=(1), keepdim=True).values.min(dim=(2), keepdim=True).values
        max_i = x_tensor.max(dim=(1), keepdim=True).values.max(dim=(2), keepdim=True).values
        x_tensor = ((x_tensor-min_i) / (max_i - min_i)) * 255
        styleGAN_transformedimage = [transforms.ToPILImage()(x_) for x_ in x_tensor.type(torch.uint8)]
</code></pre>
<h1>
<a name="get-dominant-color-2" class="anchor" href="#get-dominant-color-2"></a>##<span class="hashtag">#Get</span> Dominant Color</h1>
<pre><code>        styleGAN_transformedimage = [get_dominant_color(x) for x in styleGAN_transformedimage]
      
        Avg_color = [torch.tensor(x,dtype=float,requires_grad=False) for x in styleGAN_transformedimage]
        Avg_color = [color.to(device) for color in Avg_color]        
        Avg_color = torch.stack(Avg_color)
        
        Target_color_match = torch.cat(len(Avg_color)*[Target_color])
        Avg_color = torch.unsqueeze(Avg_color,1)
</code></pre>
<h1>
<a name="loss-calculation-3" class="anchor" href="#loss-calculation-3"></a>##<span class="hashtag">#Loss</span> calculation########</h1>
<pre><code>        Edge_loss = criterion(fakeEdge_image,sketch_image)   
        color_loss = criterion(Avg_color,Target_color_match)
        total_loss = Edge_loss + color_loss
        total_loss.backward()
        optimizer.step()
        
         
        train_loss = train_loss + ((1 / (batch_idx + 1)) * (total_loss.data - train_loss))
     # print training/validation statistics 
    print('Epoch: {} \tTraining Loss: {:.6f}'.format(epoch,train_loss))
    
    del total_loss
    del styleGan_output
    del fakeEdge_image
    del Edge_loss
  
   torch.save(model.state_dict(),f"model_history/mu_stylegan{epoch}.pth")
</code></pre>
<p>This is my entire training loop</p>
<h1>
<a name="below-steps-in-the-above-code-are-not-from-the-model-output-ie-sketch_imageavg_color-and-target_color_match-4" class="anchor" href="#below-steps-in-the-above-code-are-not-from-the-model-output-ie-sketch_imageavg_color-and-target_color_match-4"></a>Below steps in the above code are not from the model output i.e sketch_image,Avg_color and Target_color_match.</h1>
<p>Edge_loss = criterion(fakeEdge_image,sketch_image)<br>
color_loss = criterion(Avg_color,Target_color_match)</p>
<h1>
<a name="does-states-of-all-these-to-get-saved-below-5" class="anchor" href="#does-states-of-all-these-to-get-saved-below-5"></a>Does states of all these to get saved below</h1>
<p>torch.save(model.state_dict(),f"model_history/mu_stylegan{epoch}.pth")</p>
<h1>
<a name="below-is-the-way-i-am-evaluating-6" class="anchor" href="#below-is-the-way-i-am-evaluating-6"></a>Below is the way i am Evaluating:</h1>
<p>Target_color = torch.zeros(1, 1, 3, dtype=torch.float)<br>
Target_color[:,:,2] = 255<br>
Target_color_blue = Target_color.to(device)</p>
<p>model.load_state_dict(torch.load(os.path.join(“model_history/mu_stylegan178.pth”),map_location=torch.device(‘cuda’)))<br>
<span class="hashtag">#progress_bar</span> = tqdm(enumerate(trainA_dataloader,trainB_dataloader),total=len(trainA_dataloader))<br>
model.eval()<br>
outputs = []<br>
for batch_idx, img in enumerate(loaders[‘test_rgbimage_dataloader’]):<br>
if use_cuda:<br>
real = img[0].to(device)<br>
sketch_image = Cyclegan_model(real)<br>
recon = mu_stylegan(real,sketch_image,Target_color_blue.float(),noise)<br>
sketch_fake_test = recon.detach().cpu().numpy()<br>
for img in sketch_fake_test:<br>
img = img/2+0.5<br>
plt.imshow(np.transpose(img, (1, 2, 0)))<br>
plt.show()</p>

ptrblck: <p>I’m still unsure which issue we are debugging, but in any case:</p>
<pre><code class="lang-python">color_loss = criterion(Avg_color,Target_color_match)
</code></pre>
<p>won’t calculate the gradients for the model parameters, since you were recreating tensors and are thus detaching the computation graph in:</p>
<pre><code class="lang-python">Avg_color = [torch.tensor(x,dtype=float,requires_grad=False) for x in styleGAN_transformedimage]
</code></pre>

KURUVILLA_ABRAHAM: <aside class="quote no-group" data-username="KURUVILLA_ABRAHAM" data-post="8" data-topic="142514">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kuruvilla_abraham/48/45695_2.png" class="avatar"> KURUVILLA_ABRAHAM:</div>
<blockquote>
<pre><code class="lang-auto">        styleGAN_transformedimage = [get_dominant_color(x) for x in styleGAN_transformedimage]
</code></pre>
</blockquote>
</aside>
<p>Hi Patrick,</p>
<p>I converted  this as a tensor back  as i am sending the the image in PIL format to the below function(get_dominant_color)</p>
<h1><a name="code-1" class="anchor" href="#code-1"></a>Code</h1>
<p>styleGAN_transformedimage = [get_dominant_color(x) for x in styleGAN_transformedimage]</p>
<p>What way i can do this to retain it as a tensor</p>
<p>Also what “detaching the computation graph” mean ?what have to do to fix this in the step ?</p>
<pre><code class="lang-auto">Avg_color = [torch.tensor(x,dtype=float,requires_grad=False) for x in styleGAN_transformedimage]
</code></pre>

ptrblck: <p>If you are using other libraries, such as numpy or PIL, you would detach the tensor from the computation graph since Autograd isn’t able to track these operations. This would mean that previous operations are not attached to the newly created tensors in:</p>
<pre><code class="lang-python">Avg_color = [torch.tensor(x,dtype=float,requires_grad=False) for x in styleGAN_transformedimage]
</code></pre>
<p>and thus no parameters used in the previous operation will get any gradients.<br>
You would have to either use PyTorch operations or write custom <code>autograd.Function</code>s as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>

KURUVILLA_ABRAHAM: <p>How can we find dominant color of an image in pytorch from the incoming Tensor.have to remove all the white pixels outside the edges of the tensor ?.</p>
<p>The above operations needs to take place in the training loop …</p>
<p>I have built this using PIL in my above training loop</p>

ptrblck: <p>I don’t know what exactly <code>get_dominant_color</code> does, but you should be able to mask the pixels in the tensor directly.</p>

KURUVILLA_ABRAHAM: <p>Hello patrick ,domiant color means the most used color in a image out of all …</p>