My Discriminator model collapsed and always returns 1s

unnormal: <p>I composed a Discriminator nn model alike the one from the <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">DCGAN tutorial</a>. With main differences that I use biases and drop the batch normalization.</p>
<p>When training starts the model is showing a good loss optimization, but for some reason after several training batches the Discriminator suddenly curved up to the 1s producer with increased loss:</p>
<blockquote>
<p>[epoch - 0, 78/3166]<br>
Discriminator real mean: 0.995487093925, Discriminator fake mean: 0.000025399677<br>
discriminatorRealError: 0.288825571537, discriminatorFakeError: 0.001625579316<br>
[epoch - 0, 79/3166]<br>
Discriminator real mean: 1.000000000000, Discriminator fake mean: 1.000000000000<br>
discriminatorRealError: 0.000000000000, discriminatorFakeError: 64.000000000000</p>
</blockquote>
<p><em>discriminatorRealError - the loss for real data, discriminatorFakeError - the loss for ‘fake’ data</em></p>
<p><strong>After the model collapsed the final layer sigmoid always return 1, which makes all gradients in the model equal to zeros and it stops train.</strong></p>
<p>Batch normalization layers help for a while but do not fix the problem, so I purpesly dropped them to easily reproduce the issue.<br>
I use biases because I’m experimenting and don’t see reason to drop them.</p>
<p>My code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision
import numpy as np
import matplotlib.pyplot as plt
import torchvision.transforms as transforms
import pathlib
from torchinfo import summary

imageWidth = 178
imageHeight = 218
batch_size = 64
path = "C:\\Users\\PC\\Downloads\\celeba"
netPath = pathlib.Path(__file__).parent.resolve()

class DiscriminatorNet:
    def __init__(self):
        self.model = nn.Sequential(
            nn.Conv2d(3, 24, 5, stride=2, bias=True),
            nn.LeakyReLU(),
            nn.Conv2d(24, 24, 5, stride=2, bias=True),
            nn.LeakyReLU(),
            nn.Conv2d(24, 36, 5, stride=2, bias=True),
            nn.LeakyReLU(),
            nn.Conv2d(36, 68, 5, stride=3, bias=True),
            nn.LeakyReLU(),
            nn.Conv2d(68, 1, kernel_size=(7, 5), bias=True),
            nn.Sigmoid()
        )
    
    def forward(self, input):
        return self.model(input)

if __name__ == '__main__':
    device = torch.device("cuda:0")

    discriminator = DiscriminatorNet()
    discriminator.model.to(device)

    # summary(discriminator.model, input_size=(batch_size, 3, imageHeight, imageWidth))
    # exit()

    discriminator.model.load_state_dict(torch.load(netPath / 'discriminatorBefore.pth', weights_only=True))
    print(f'loaded from {netPath}')

    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
    dataset = torchvision.datasets.ImageFolder(root=path, transform=transform)

    dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                             shuffle=False, num_workers=2)

    train = True

    if train == True:
        discriminatorOpt = torch.optim.Adam(discriminator.model.parameters(), betas=(0.5, 0.999))

        for epoch in range(5):
            for i, batch in enumerate(dataloader):
                inputs = batch[0].to(device)

                # labels
                trueLabels = torch.full((inputs.size(0),), 1, dtype=torch.float32, device=device)
                fakeLabels = torch.full((inputs.size(0),), 0, dtype=torch.float32, device=device)

                discriminatorOpt.zero_grad()

                # calculating discriminator loss for the real data
                discriminatorRealsOutput = discriminator.model(inputs).view(-1)
                discriminatorRealError = torch.abs(discriminatorRealsOutput - trueLabels).sum()
                discriminatorRealError.backward()
                
                # calculating discriminator loss for the 'fake' data
                noise = torch.rand(inputs.size(0), 3, imageHeight, imageWidth, device=device)
                discriminatorFakeOutputs = discriminator.model(noise).view(-1)
                discriminatorFakeError = torch.abs(discriminatorFakeOutputs - fakeLabels).sum()
                discriminatorFakeError.backward()

                #gradD = discriminator.model[-2].weight.grad
                discriminatorOpt.step()

                print(f'[epoch - {epoch}, {i}/{len(dataloader)}]')
                print(f'Discriminator real mean: {discriminatorRealsOutput.view(-1).mean():.12f}, Discriminator fake mean: {discriminatorFakeOutputs.view(-1).mean():.12f}')
                print(f'discriminatorRealError: {discriminatorRealError:.12f}, discriminatorFakeError: {discriminatorFakeError:.12f}')
            
</code></pre>
<p>This behavior isn’t stably reproducing and depend what weights and biases <code>PyTorch</code> generated for the Discriminator while initializing. This is what the discriminator model looks like in my example <a href="https://limewire.com/d/mMiVN#FclEbNMUnC" rel="noopener nofollow ugc">before and after the training</a>.</p>
<p>It would be nice to know how can I profile this or to hear that I’m doing something fundamentally wrong. The model is trying to outsmart me I don’t like that.</p>