UNET multiple outputs: channels vs separate layers

Nolan241: <p>hi,</p>
<p>i’m trying to build a network which gets an image as input<br>
and outputs different angle depth map views for the input.</p>
<p>like here: <a href="https://github.com/happylun/SketchModeling" rel="noopener nofollow ugc">SketchModeling</a></p>
<p>so far i have:<br>
o Encoder (Down half of an Unet)<br>
o Decoder (Up half of an Unet)</p>
<p>so the encoder takes the image and for each view there is a decoder, which gets the encoder output, skip_connections as input.</p>
<p>out, skip = encoder(image)<br>
view1_out = view1_decoder(out,skip)<br>
view2_out = view2_decoder(out,skip)</p>
<p>my questions now is, if i have multiple decoder outputs, do i use them as channel or as extra conv2d?</p>
<p>my current outputs are:</p>
<p>o binary mask (1 channel, 0 or 1, float)<br>
o depth image (1 channel, 0 - 1, float)<br>
o depth exr-format (1 channel, 0 - 10, float)<br>
o normal map (3 channels, 0 - 1, float)</p>
<p>i could join them together as channels:</p>
<p>decoder_out = 64<br>
final_out = 6 (all out channels joined)</p>
<p>final_conv = Conv2d(decoder_out, final_out , kernel_size=1)</p>
<p>or have an extra conv2d for each one of them:</p>
<p>final_convs = ModuleList()<br>
final_convs.append(Conv2d(decoder_out, 1, kernel_size=1))<br>
…<br>
final_convs.append(Conv2d(decoder_out, 3, kernel_size=1))</p>
<p>which approach would be better for training and grads?</p>
<p>greetings and happy holydays<br>
nolan</p>

J_Johnson: <p>Splitting your Conv2d on the final output layer for different output channels would just be semantics and not change the math(however, it would slow down the forward pass).</p>
<p>Let me prove it:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

torch.manual_seed(0)

batch_size=2
in_channels=3
out_channels=2
kernel=3

#making image and copy
x = torch.rand(batch_size,in_channels,4,4)
y = x.clone()

#instantiate the weights
weights = torch.rand(out_channels, in_channels, kernel,kernel)

#initial path, 1 conv2d to rule them all
output=F.conv2d(x, weights, padding=1)

#split path
output1=F.conv2d(x, weights[:1,:,:,:], padding=1)
output2=F.conv2d(y, weights[-1:,:,:,:], padding=1)
output1_2=torch.cat([output1,output2], dim=1)

#confirm they are identical
print(torch.allclose(output, output1_2))
</code></pre>

Nolan241: <p>hey,</p>
<p>thanks for the reply. thats what i throught, but was not sure.</p>
<p>in the project mentioned above, they calculate the loss and check if the result against a discriminator.</p>
<p>so far i calculate the loss like this:</p>
<p>target and output = (B,6,H,W)<br>
gen_criterion = L1Loss()<br>
disc_criterion = BCELoss()<br>
alpha = 1.0<br>
beta = 0.01</p>
<p>the discriminator currently implented takes the stack ouf view outputs and returns<br>
range 0-1</p>
<h1><a name="train-disc-1" class="anchor" href="#train-disc-1"></a>train disc</h1>
<p>output = generator(input).detach()</p>
<p>out_real = discriminator(target)<br>
out_fake = discriminator(output)</p>
<p>loss_real = disc_criterion(out_real, torch.ones((B)))<br>
loss_fake = disc_criterion(out_fake, torch.zeros((B)))<br>
disc_loss = loss_real + loss_fake</p>
<p>disc_optim.zero_grad()<br>
disc_loss.backward()<br>
disc_optim.step()</p>
<h1><a name="train-gen-2" class="anchor" href="#train-gen-2"></a>train gen</h1>
<p>output = generator(input)<br>
out_fake = discriminator(output)<br>
loss_gen = gen_criterion(output ,target)<br>
loss_disc = disc_criterion(out_fake, torch.ones((B)))</p>
<p>total_loss = loss_gen * alpha + loss_disc * beta</p>
<p>gen_optim.zero_grad()<br>
total_loss.backward()<br>
gen_optim.step()</p>
<p>typical GAN setup, but i dont think thats the right way.<br>
in the other project they somehow subtract the inverted negative loss of the discriminator<br>
from the loss of the generator to reward it for ‘REAL’ outputs.</p>
<p>so if it is a total fake use the full generator loss, but if it seems real decrement the loss.</p>
<p>total fake = 0<br>
seems real = 1</p>
<p>total_loss = loss_gen - loss_disc</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c83676b9f27cdfef000f51e89659fbbf2284af7a.gif" data-download-href="https://discuss.pytorch.org/uploads/default/c83676b9f27cdfef000f51e89659fbbf2284af7a" title="loss_gif"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c83676b9f27cdfef000f51e89659fbbf2284af7a.gif" alt="loss_gif" data-base62-sha1="szabikNaCaxhjz9LG6lmtSFgcIi" width="657" height="500" data-dominant-color="E2E0E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_gif</span><span class="informations">664×505 34.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>they used tensorflow, but how to implement it in pytorch?</p>
<p>greetings<br>
nolan</p>