Can't Train a Network to do an identity operation over a single image

username: Michael_Lellouch
cooked: <p>Hey everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I’m trying to train a network to do an image processing operation. Since I was having some trouble with that, I moved to just trying to train a network to do nothing, and it still doesn’t converge to anything good.</p>
<p>Does anything looks off in my code? Any idea/remark will be gladly appreciated</p>
<p>Thanks!</p>
<pre><code class="lang-auto"> class Net(nn.Module):
        def __init__(self):
            super().__init__()
            self.conv1 = nn.Conv2d(3, 64, 3, padding=1)
            self.conv2 = nn.Conv2d(64, 128, 3, padding=1)
            self.conv3 = nn.Conv2d(128, 256, 3, padding=1)
            self.conv4 = nn.Conv2d(256, 256, 3, padding=1)
            self.conv5 = nn.Conv2d(256, 256, 3, padding=1)
            self.conv6 = nn.Conv2d(256, 128, 3, padding=1)
            self.conv7 = nn.Conv2d(128, 64, 3, padding=1)
            self.conv8 = nn.Conv2d(64, 3, 3, padding=1)
    
    
        def forward(self, x):
            x = F.relu(self.conv1(x))
            x = F.relu(self.conv2(x))
            x = F.relu(self.conv3(x))
            x = F.relu(self.conv4(x))
            x = F.relu(self.conv5(x))
            x = F.relu(self.conv6(x))
            x = F.relu(self.conv7(x))
            x = F.relu(self.conv8(x))
            return x
    
    class SingleImageDataset(Dataset):
    
        def __init__(self, path):
            self.image = Image.open(path)
            tensor_trans = transforms.ToTensor()
            tens = tensor_trans(self.image)
            self.stds = [tens[i].std().item() for i in range(3)]
            self.means = [tens[i].mean().item() for i in range(3)]
            normalize = transforms.Normalize(mean=self.means, std=self.stds)
            self.final_data = normalize(tens)
    
        def __len__(self):
          return 1
    
        def __getitem__(self, idx):
          return self.final_data
    
    naruto_dataset = SingleImageDataset('./image/naruto.jpg')
    naruto_loader = torch.utils.data.DataLoader(naruto_dataset, batch_size=1,
                                              shuffle=False, num_workers=1)
    net = Net()
    net.to(device)
    criterion = nn.L1Loss()
    optimizer = optim.Adam(net.parameters(), lr=0.001)
    
    losses = []
    for epoch in tqdm.tqdm(range(500)):
        running_loss = 0.0
        for i, data in enumerate(naruto_loader, 0):
            inputs = data.cuda()
            # zero the parameter gradients
            optimizer.zero_grad()
    
            # forward + backward + optimize
            outputs = net(inputs)
            loss = criterion(outputs, inputs)
            losses.append(loss.item())
            # print(loss.item())
            loss.backward()
            optimizer.step()
</code></pre>
<p>And the loss converges on 0.4 (which is horrible, since I’m using L1Loss which is a mean)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33f1aee4e9548c9d6bd225e2f4d12109bc285670.png" alt="download" data-base62-sha1="7pw9B7Fjm42jYcqzCOT7JGHWw6s" width="372" height="248"></p>

username: tom
cooked: <p>What is your input range? I’m asking because usually people leave out the final relu.</p>

username: Michael_Lellouch
cooked: <p>Nice catch!<br>
Indeed my input was also negative, so applying ReLUs all the way was bound to give me problems.<br>
The loss now is around 0.04, do you think this is still a problem or acceptable?</p>

username: tom
cooked: <p>I wouldn’t know what to say about the number.<br>
You could look at your input and output images or go back to the original task to see.<br>
The other bit that I do know is that e.g. the <a href="https://github.com/pytorch/examples/tree/master/fast_neural_style" rel="noopener nofollow ugc">PyTorch style transfer example</a> has normalization layers in between. But you will know best what is applicable to your task.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Michael_Lellouch
cooked: <p>Super helpful resource, Thanks a bunch!<br>
Michael</p>