How to get continuous value from output layer

akib62: <p>I have a code (from <a href="https://medium.com/@nutanbhogendrasharma/pytorch-convolutional-neural-network-with-mnist-dataset-4e8a4265e118" rel="noopener nofollow ugc">here</a>) to classify the MINST digits. The code is working fine. Here they used <code>CrossEntropyLoss</code> and <code>Adam</code> optimizer.</p>
<p>The model code is given below</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Sequential(         
            nn.Conv2d(
                in_channels=1,              
                out_channels=16,            
                kernel_size=5,              
                stride=1,                   
                padding=2,                  
            ),                              
            nn.ReLU(),                      
            nn.MaxPool2d(kernel_size=2),    
        )
        self.conv2 = nn.Sequential(         
            nn.Conv2d(16, 32, 5, 1, 2),     
            nn.ReLU(),                      
            nn.MaxPool2d(2),                
        )
        # fully connected layer, output 10 classes
        self.out = nn.Linear(32 * 7 * 7, 10)
        # self.softmax = torch.nn.Softmax(dim=1)
    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        # flatten the output of conv2 to (batch_size, 32 * 7 * 7)
        x = x.view(x.size(0), -1)       
        output = self.out(x)
        # output = self.softmax(output)
        return output, x    # return x for visualization

</code></pre>
<p>Now, I wanted to get continuous value from the output layer. Say, I want the output as alike i.e, 1.0, 0.9, 8.6, 7.0, etc. If the value of the output layer is 1.0 and the label is 1 that means the prediction is perfect. Otherwise, not perfect. More simply, I want to think the MNIST digits as a regression problem.</p>
<p>So, I changed the loss function to <code>MSELoss</code> and optimizer to <code>SGD</code> (the rest of the code remains as the same as the website). But now, I am getting an error</p>
<pre><code class="lang-auto">/home/Opps_0/.local/lib/python3.8/site-packages/torch/nn/modules/loss.py:528: UserWarning: Using a target size (torch.Size([100])) that is different to the input size (torch.Size([100, 10])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/Opps_0/Desktop/MNIST/src/train.py", line 60, in &lt;module&gt;
    train(NB_EPOCS, model, loaders)
  File "/home/Opps_0/Desktop/MNIST/src/train.py", line 45, in train
    loss = criterion(output, b_y)
  File "/home/Opps_0/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/Opps_0/.local/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 528, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "/home/Opps_0/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 2925, in mse_loss
    expanded_input, expanded_target = torch.broadcast_tensors(input, target)
  File "/home/Opps_0/.local/lib/python3.8/site-packages/torch/functional.py", line 74, in broadcast_tensors
    return _VF.broadcast_tensors(tensors)  # type: ignore
RuntimeError: The size of tensor a (10) must match the size of tensor b (100) at non-singleton dimension 1
</code></pre>
<p>Could you tell me what I have to change to get the continuous value for the output layer?</p>

ptrblck: <p>I’m not sure how the target <code>b_y</code> is created, but it seems that it has a shape of <code>[unknown, 100]</code>, which doesn’t fit the model output of <code>[batch_size, 10]</code>. Could you explain how the target is computed and what its dimensions/shapes represent?</p>

akib62: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> the data loader is</p>
<pre><code class="lang-auto">def data_loaders():
    train_data = datasets.MNIST(
        root = 'data',
        train = True,                         
        transform = transforms.ToTensor(), 
        download = True,            
    )
    test_data = datasets.MNIST(
        root = 'data', 
        train = False, 
        transform = transforms.ToTensor()
    )

    train_test_loaders = {
        'train' : torch.utils.data.DataLoader(train_data, 
                                            batch_size=100, 
                                            shuffle=True, 
                                            num_workers=1),
        
        'test'  : torch.utils.data.DataLoader(test_data, 
                                            batch_size=100, 
                                            shuffle=True, 
                                            num_workers=1),
    }
    return train_test_loaders
</code></pre>
<p>Later, at the time of training</p>
<pre><code class="lang-auto">def train(NB_EPOCS, model, loaders):
    model.train()
        
    # Train the model
    total_step = len(loaders['train'])
    for epoch in range(NB_EPOCS):
        for i, (images, labels) in enumerate(loaders['train']):
            b_x = Variable(images)   # batch x
            b_y = Variable(labels)   # batch y
            print(np.shape(b_x), np.shape(b_y))
            output = model(b_x)[0]               
            loss = criterion(output, b_y)
</code></pre>
<p>The shape of the <code>b_x</code> and <code>b_y</code> is</p>
<pre><code class="lang-auto">torch.Size([100, 1, 28, 28]) torch.Size([100])
</code></pre>

ptrblck: <p>Thanks for the update! I cannot reproduce the issue using your code and the training works fine:</p>
<pre><code class="lang-python">model = CNN()
loaders = data_loaders()
    
# Train the model
criterion = nn.CrossEntropyLoss()

for i, (images, labels) in enumerate(loaders['train']):
    b_x = images   # batch x
    b_y = labels  # batch y
    print(np.shape(b_x), np.shape(b_y))
    output = model(b_x)[0]               
    loss = criterion(output, b_y)
</code></pre>

akib62: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> are you using <code>MSELoos</code> or <code>CrossEntropy</code>? CrossEntropy is working perfectly but getting error at the time of <code>MSELoos</code></p>

ptrblck: <p>As seen in my code snippet I’m using <code>nn.CrossEntropyLoss</code> since it’s a classification use case.<br>
If you want to use <code>nn.MSELoss</code> for specific reasons, you would have to provide the target in the same shape as the model output, i.e. <code>[batch_size, 10]</code> e.g. by one-hot encoding it.</p>

akib62: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your reply. But I already tried one hot coding and getting different error</p>
<p>You can see the <code>dtype</code> of the <code>b_y</code> is <code>Int64</code> and I am getting an error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/Opps_0/Desktop/MNIST/src/train.py", line 69, in &lt;module&gt;
    train(NB_EPOCS, model, loaders)
  File "/home/Opps_0/Desktop/MNIST/src/train.py", line 60, in train
    loss.backward()    
  File "/home/Opps_0/.local/lib/python3.8/site-packages/torch/tensor.py", line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/Opps_0/.local/lib/python3.8/site-packages/torch/autograd/__init__.py", line 145, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Found dtype Long but expected Float
</code></pre>
<p>Code</p>
<pre><code class="lang-auto">def train(NB_EPOCS, model, loaders):
    model.train()
    # Train the model
    total_step = len(loaders['train'])
    for epoch in range(NB_EPOCS):
        for i, (images, labels) in enumerate(loaders['train']):
            # gives batch data, normalize x when iterate train_loader
            b_x = Variable(images)   # batch x
            b_y = Variable(labels)   # batch y
            print(np.shape(b_x), b_x.dtype, np.shape(b_y), b_y.dtype)
            ohe_target = torch.nn.functional.one_hot(b_y, num_classes=10)
            output = model(b_x)[0]   
            loss = criterion(output, ohe_target)
</code></pre>
<pre><code class="lang-auto">torch.Size([100, 1, 28, 28]) torch.float32 torch.Size([100]) torch.int64
</code></pre>
<p>If I change the dtype of the b_y to <code>Float</code> by the following code</p>
<pre><code class="lang-auto">b_y=b_y.float()
</code></pre>
<p>Then I am getting another error,</p>
<pre><code class="lang-auto">RuntimeError: one_hot is only applicable to index tensor.
</code></pre>
<p>What I have to do to solve the errors? (Main <a href="https://discuss.pytorch.org/t/runtimeerror-found-dtype-long-but-expected-float/132086">link</a> of the question)</p>

ptrblck: <p>I guess you are transforming <code>b_y</code> to <code>float</code> before applying <code>one_hot</code>, not after?<br>
This works:</p>
<pre><code class="lang-python"># Train the model
criterion = nn.MSELoss()

for i, (images, labels) in enumerate(loaders['train']):
    b_x = images   # batch x
    b_y = labels  # batch y
    ohe_target = torch.nn.functional.one_hot(b_y, num_classes=10).float()
    output = model(b_x)[0]               
    loss = criterion(output, ohe_target)
</code></pre>