Problem while training model

dato_nefaridze: <p>i am training model on cats vs dogs dataset.images are 50x50x3, then i view them to 3x50x50 and train model.loss funciton is MSE, optimizer Adam, and actiovation function relu.it also has convolutional layers.model returns 2d vector which is calculated by softmax function(it is from functional).it is not learning at all, i mean on test set it outputs same values all the time, what the problem might be?</p>

beaupreda: <p>Hi,</p>
<p>If you could post code to reproduce your problem, it would be easier for us to find the problem.</p>
<p>Also, with images, when you want to change the order of the dimensions, you should use the <code>permute</code> function instead of the <code>view</code> function. This small example shows the difference between the two:</p>
<pre><code class="lang-auto">    x = torch.rand(size=(4, 4, 3), dtype=torch.float32)
    print(x)
    # tensor([[[0.0136, 0.6666, 0.5414],
    #          [0.8499, 0.6427, 0.8142],
    #          [0.9565, 0.4752, 0.0079],
    #          [0.5476, 0.2150, 0.9677]],
    # 
    #         [[0.9483, 0.6250, 0.1009],
    #          [0.8543, 0.1606, 0.0530],
    #          [0.7296, 0.6276, 0.1459],
    #          [0.0894, 0.5672, 0.2201]],
    # 
    #         [[0.1019, 0.4461, 0.1278],
    #          [0.6479, 0.5163, 0.2692],
    #          [0.3704, 0.4845, 0.1759],
    #          [0.2597, 0.6056, 0.2702]],
    # 
    #         [[0.4689, 0.0948, 0.3193],
    #          [0.0574, 0.1467, 0.2993],
    #          [0.8013, 0.7025, 0.2048],
    #          [0.0281, 0.9036, 0.0024]]])
    y = x.view(3, 4, 4)
    z = x.permute(dims=(2, 0, 1))
    print(y)
    # tensor([[[0.0136, 0.6666, 0.5414, 0.8499],
    #          [0.6427, 0.8142, 0.9565, 0.4752],
    #          [0.0079, 0.5476, 0.2150, 0.9677],
    #          [0.9483, 0.6250, 0.1009, 0.8543]],
    # 
    #         [[0.1606, 0.0530, 0.7296, 0.6276],
    #          [0.1459, 0.0894, 0.5672, 0.2201],
    #          [0.1019, 0.4461, 0.1278, 0.6479],
    #          [0.5163, 0.2692, 0.3704, 0.4845]],
    # 
    #         [[0.1759, 0.2597, 0.6056, 0.2702],
    #          [0.4689, 0.0948, 0.3193, 0.0574],
    #          [0.1467, 0.2993, 0.8013, 0.7025],
    #          [0.2048, 0.0281, 0.9036, 0.0024]]])
    print(z)
    # tensor([[[0.0136, 0.8499, 0.9565, 0.5476],
    #          [0.9483, 0.8543, 0.7296, 0.0894],
    #          [0.1019, 0.6479, 0.3704, 0.2597],
    #          [0.4689, 0.0574, 0.8013, 0.0281]],
    # 
    #         [[0.6666, 0.6427, 0.4752, 0.2150],
    #          [0.6250, 0.1606, 0.6276, 0.5672],
    #          [0.4461, 0.5163, 0.4845, 0.6056],
    #          [0.0948, 0.1467, 0.7025, 0.9036]],
    # 
    #         [[0.5414, 0.8142, 0.0079, 0.9677],
    #          [0.1009, 0.0530, 0.1459, 0.2201],
    #          [0.1278, 0.2692, 0.1759, 0.2702],
    #          [0.3193, 0.2993, 0.2048, 0.0024]]])
</code></pre>

dato_nefaridze: <p>thanks, so this is code:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1=nn.Conv2d(3,32,5)
        self.conv2=nn.Conv2d(32,64,5)
        self.conv3=nn.Conv2d(64,128,5)
        self.fc1=nn.Linear(128*38*38,100)
        self.fc2=nn.Linear(100,2)
    def forward(self, x):
      x=F.relu(self.conv1(x))
      x=F.relu(self.conv2(x))
      x=F.relu(self.conv3(x))
      x=x.view(-1,128*38*38)
      x=F.relu(self.fc1(x))
      x=self.fc2(x)
      return torch.sigmoid(x)  
</code></pre>
<p>this is my training:</p>
<pre><code class="lang-auto">BATCH_SIZE =1
EPOCHS = 5
optimizer = optim.Adam(net.parameters(), lr=0.001)
criterion=nn.CrossEntropyLoss()
for epoch in range(EPOCHS):
    for i in tqdm(range(0, len(train_X), BATCH_SIZE)): 
        batch_X = train_X[i:i+BATCH_SIZE].permute(dims=(0,3, 1, 2))
        batch_y = train_y[i:i+BATCH_SIZE]
        batch_y=[x[0] for x in batch_y]
        batch_y=(torch.Tensor(batch_y).type(torch.LongTensor)).to(device)
        net.zero_grad()
        outputs = net(batch_X)
        loss = criterion(outputs, batch_y)
        loss.backward()
        optimizer.step()
</code></pre>
<p>additional information.above X,Y is:</p>
<pre><code class="lang-auto">X = torch.Tensor([i[0] for i in train_data])

X = X/255.0

y = torch.Tensor([i[1] for i in train_data])

X=X.to(device)

y=y.to(device)
</code></pre>

dato_nefaridze: <p>i checked and sigmoid is compatible with cross entropy, also i corrected code, according to your suggestion(use permute), i used permute to pass whole batch to net, i keep first dimention the same and changing only last(bebcause input of network should be 3x50x50)</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects logits, not sigmoid outputs, so you should remove this activation layer.</p>

dato_nefaridze: <p>ok thanks, it is my fault after many failures i decidet to copy entire network from this guy:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/467e3a6791250ea9bd1e39b50cf83bd3aa6ef199.png" data-download-href="https://discuss.pytorch.org/uploads/default/467e3a6791250ea9bd1e39b50cf83bd3aa6ef199" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/467e3a6791250ea9bd1e39b50cf83bd3aa6ef199_2_690x387.png" alt="image" data-base62-sha1="a3BREU9bKj0iRgunPiSmWp5TQZb" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/467e3a6791250ea9bd1e39b50cf83bd3aa6ef199_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/467e3a6791250ea9bd1e39b50cf83bd3aa6ef199_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/6/467e3a6791250ea9bd1e39b50cf83bd3aa6ef199.png 2x" data-dominant-color="F2F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1366×768 74.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>  seems he uploads incorrect projects.</p>

dato_nefaridze: <p>but i have got same result <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:">  by removing sigmoid,thanks anyway.</p>

ptrblck: <p>Or the code uses <code>nn.BCELoss</code>. Could you check that?</p>

dato_nefaridze: <p>no it he used crossentropy here is proof:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/933f51e07dda2302ad56f59f30c3c298b5c5668f.png" data-download-href="https://discuss.pytorch.org/uploads/default/933f51e07dda2302ad56f59f30c3c298b5c5668f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/933f51e07dda2302ad56f59f30c3c298b5c5668f.png" alt="image" data-base62-sha1="l0BQIr6jEWrn6pfSqzO4HSbyGhF" width="690" height="387" data-dominant-color="F6F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1366×768 27.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Thanks for clarification.<br>
In that case, the model <em>might</em> somehoe work, but model outputs are not in an expected range and thus you might see some side effects, e.g. vanishing gradients.</p>