CrossEntropyLoss target size

kdeary: <p>Beginner here. I am having trouble with target size and how to reshape for CrossEntropyLoss.</p>
<p>I received this error: <code>**RuntimeError** : Expected target size [100, 4], got [100].</code></p>
<p>I’ve seen that many have had this same issue, but still do not understand the other posts. How do I reshape?</p>
<pre><code class="lang-auto"># define model
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.layer1 = nn.Linear(28, 64)
        self.layer2 = nn.ReLU()
        self.layer3 = nn.Linear(64, 32)
        self.layer4 = nn.ReLU()
        self.layer5 = nn.Linear(32, 4) # 4 classes
        
    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)
        x = self.layer5(x)        
        return x

model = Model()

# loss &amp; optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.1)

#train model
num_epochs = 5

for epoch in range(num_epochs):
    for i, data in enumerate(train_loader):
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward propagation
        outputs = model(inputs)
        
        loss = criterion(outputs, labels)

        # backward propagation &amp; optimize
        loss.backward()
        optimizer.step()

# shapes after layers
Shape after layer: torch.Size([100, 28, 64])
Shape after layer: torch.Size([100, 28, 64])
Shape after layer: torch.Size([100, 28, 32])
Shape after layer: torch.Size([100, 28, 32])
Shape after layer: torch.Size([100, 28, 4])
</code></pre>

ptrblck: <p>Based on the posted shapes your output has the shape <code>[batch_size=100, nb_classes=28, seq_len=4]</code>, which requires the target to have the shape <code>[batch_size=100, seq_len=4]</code>.<br>
However, based on your model architecture I assume you want to use 4 classes in the prediction.<br>
Since your output is 3-dimensional: are you using a sequence of samples or is the input reshaped in a wrong way?</p>

J_Johnson: <p>You’re running images through Linear layers.</p>
<ol>
<li>Linear layers perform poorly on images and Conv2d layers are more sufficient in order to distill 2d patterns into simpler data. For example, see here: <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Training a Classifier — PyTorch Tutorials 1.13.0+cu117 documentation</a>
</li>
<li>You’re ignoring one of the dims of your images. If you <em>must</em> put it through linear layers, try flattening your tensor before putting it into the model. This will require changing your layer sizes, in particular, the first layer.</li>
</ol>
<p><code>inputs = inputs.view(-1, 28*28)</code><br>
<code>self.layer1 = nn.Linear(28*28, 64)</code></p>

kdeary: <p>The data is based on 28x28 greyscale images, but not the images itself.</p>
<p>From this code:</p>
<pre><code class="lang-auto">example = enumerate(test_loader)
batch_idx, (example_data, example_targets) = next(example)
print(example_data.shape)
</code></pre>
<p><code>torch.Size([1000, 28, 28])</code></p>

ptrblck: <p><a class="mention" href="/u/j_johnson">@J_Johnson</a>’s suggestion of flattening the input sounds reasonable so did you try to fix the issue with it?</p>

kdeary: <p>It did work! I am able to train the network.</p>
<p>My accuracy is currently at 0.32. Do you have any tips to increase the accuracy?</p>

J_Johnson: <p>Here are the top model architecture scores for MNIST Fashion dataset classification with links to papers.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://paperswithcode.com/sota/image-classification-on-fashion-mnist">
  <header class="source">

      <a href="https://paperswithcode.com/sota/image-classification-on-fashion-mnist" target="_blank" rel="noopener nofollow ugc">paperswithcode.com</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/365;"><img src="https://production-media.paperswithcode.com/sota-thumbs/image-classification-on-fashion-mnist-large_e13121e2.png" class="thumbnail" width="690" height="365"></div>

<h3><a href="https://paperswithcode.com/sota/image-classification-on-fashion-mnist" target="_blank" rel="noopener nofollow ugc">Papers with Code - Fashion-MNIST Benchmark (Image Classification)</a></h3>

  <p>The current state-of-the-art on Fashion-MNIST is Fine-Tuning DARTS. See a full comparison of 14 papers with code.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


CDog: <p>Hello there. I have seen a lot of others with this issue. Any advice would be appreciated. I am trying to train a GPT-2 model to take in a tokenized/padded input and predict the output. My batch size is 32. My max length is 343. I believe that the 768 comes from the model. I cannot get the loss function to work properly though. The training loop keeps throwing me errors like this:</p>
<pre><code class="lang-auto">RuntimeError: Expected target size [32, 768], got [32, 343]
</code></pre>
<p>This is my training loop so far.</p>
<pre><code class="lang-auto"># Define the model architecture
model = transformers.GPT2Model.from_pretrained('gpt2').to(device)

# Define the loss function
loss_function = nn.CrossEntropyLoss(ignore_index=0, reduction='mean')

# Define the optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

# Set the model to training mode
model.train()
print(f"input_tensors.shape before the loop: {input_tensors.shape}")
print(f"output_tensors.shape before the loop: {output_tensors.shape}")

# Loop over the number of epochs
for epoch in range(num_epochs):
    # Initialize the epoch loss
    epoch_loss = 0
    
    # Loop over the data in the dataloader
    for input_tensors, output_tensors in dataloader:
        # Send the input and target tensors to the device
        input_tensors = input_tensors.to(device)
        output_tensors = output_tensors.to(device)
        # Zero gradients
        optimizer.zero_grad()
        
        # Begin Forward pass
        logits = model(input_tensors)[0]
        
        print(f"logits.shape: {logits.shape}")
        print(f"input_tensors.shape: {input_tensors.shape}")
        print(f"output_tensors.shape: {output_tensors.shape}")
        
        # Compute the loss
        loss = loss_function(logits, output_tensors)

        # Backward pass
        loss.backward()

        # Update the model parameters
        optimizer.step()

        # Add the loss to the epoch loss
        epoch_loss += loss.item()
        # Print the epoch loss
    print(f'Epoch {epoch+1}: Loss = {epoch_loss}')
</code></pre>
<p>The corresponding sizes:</p>
<pre><code class="lang-auto">input_tensors.shape before the loop: torch.Size([2625, 343])
output_tensors.shape before the loop: torch.Size([2625, 343])
logits.shape: torch.Size([32, 343, 768])
input_tensors.shape: torch.Size([32, 343])
output_tensors.shape: torch.Size([32, 343])
</code></pre>

J_Johnson: <p>If you’re having a specific problem with a module, best to start a new thread.</p>
<p>However, GPT2Model is a Huggingface model and you should refer to their documentation to find the correct solution.</p>
<p>You can find the documentation for GPT2Model here:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/docs/transformers/model_doc/gpt2">
  <header class="source">

      <a href="https://huggingface.co/docs/transformers/model_doc/gpt2" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/372;"><img src="https://huggingface.co/front/thumbnails/docs/transformers.png" class="thumbnail" width="690" height="372"></div>

<h3><a href="https://huggingface.co/docs/transformers/model_doc/gpt2" target="_blank" rel="noopener nofollow ugc">OpenAI GPT2</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>And here is a training tutorial using the GPT2Model:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/blog/codeparrot">
  <header class="source">

      <a href="https://huggingface.co/blog/codeparrot" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:561/248;"><img src="https://huggingface.co/blog/assets/40_codeparrot/thumbnail.png" class="thumbnail" width="561" height="248"></div>

<h3><a href="https://huggingface.co/blog/codeparrot" target="_blank" rel="noopener nofollow ugc">Training CodeParrot 🦜 from Scratch</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Additionally, they have a discussion area here for asking questions related to their Transformer models:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://discuss.huggingface.co/">
  <header class="source">
      <img src="https://global.discourse-cdn.com/business7/uploads/hellohellohello/optimized/1X/67a2c0590affeba7880ebeb46a115d863972d8ba_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://discuss.huggingface.co/" target="_blank" rel="noopener nofollow ugc">Hugging Face Forums</a>
  </header>

  <article class="onebox-body">
    <img src="https://global.discourse-cdn.com/business7/uploads/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://discuss.huggingface.co/" target="_blank" rel="noopener nofollow ugc">Hugging Face Forums</a></h3>

  <p>Community Discussion, powered by Hugging Face &lt;3</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>You are running into the same issue as described in <a href="https://discuss.pytorch.org/t/crossentropyloss-target-size/167600/2">my previous post</a>. <code>nn.CrossEntropyLoss</code> expects logits in the shape <code>[batch_size, nb_classes, *]</code> and targets in the shape <code>[batch_size, *]</code> containing class indices in the range <code>[0, nb_classes-1]</code> where <code>*</code> denotes additional dimensions.<br>
Your current logits in the shape <code>[32, 343, 768]</code> correspond to <code>batch_size=32</code>, <code>nb_classes=343</code>, <code>seq_len=768</code> which doesn’t match the expected target shape.<br>
Assuming you are dealing with <code>768</code> classes <code>.permute</code> the model output to have the shape <code>[32, 768, 343]</code> via <code>logits = logits.permute(0, 2, 1)</code> and it should work.</p>

CDog: <p>Hi. Thanks for the response! I think that solution brings one step closer to training the model. Here is my new error. It bounces from 4868 to 1330 to 1640 etc… randomly. So I posted 2 examples.</p>
<pre><code class="lang-auto">return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
IndexError: Target 4868 is out of bounds.
</code></pre>
<p><code>IndexError: Target 1330 is out of bounds.</code></p>
<p>Updated code:</p>
<pre><code class="lang-auto"># Loop over the number of epochs
for epoch in range(num_epochs):
    # Initialize the epoch loss
    epoch_loss = 0
    
    # Loop over the data in the dataloader
    for input_tensors, output_tensors in dataloader:
        # Send the input and target tensors to the device
        input_tensors = input_tensors.to(device)
        output_tensors = output_tensors.type(torch.LongTensor)
        output_tensors = output_tensors.to(device)
        # Zero gradients
        optimizer.zero_grad()
        
        # Begin Forward pass
        logits = model(input_tensors)[0]
        
        print(f"logits.shape: {logits.shape}")
        print(f"input_tensors.shape: {input_tensors.shape}")
        print(f"output_tensors.shape: {output_tensors.shape}")
        logits = logits.permute(0, 2, 1)
        # Compute the loss
        print(f"logits.shape after permute: {logits.shape}")
        loss = loss_function(logits, output_tensors)


        # Backward pass
        loss.backward()

        # Update the model parameters
        optimizer.step()

        # Add the loss to the epoch loss
        epoch_loss += loss.item()
        # Print the epoch loss
    print(f'Epoch {epoch+1}: Loss = {epoch_loss}')
</code></pre>
<p>We can see the shape does change after the permute.</p>
<pre><code class="lang-auto">logits.shape: torch.Size([32, 343, 768])
input_tensors.shape: torch.Size([32, 343])
output_tensors.shape: torch.Size([32, 343])
logits.shape after permute: torch.Size([32, 768, 343])
</code></pre>
<p>If you have any ideas on what could be going wrong I would love to hear them. I will keep digging until then. Thanks again.</p>
<p>I also get these warnings everytime I run. Not sure if they are related to my error but I will dig on these as well. I don’t think they are related to PyTorch but rather TensorFlow so I don’t expect a solution here. I DO have a 2080ti on my machine.</p>
<pre><code class="lang-auto"> W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
</code></pre>

ptrblck: <p>Your permuted logits in the shape <code>[batch_size=32, nb_classes=768, seq_len=343]</code> contain values for <code>768</code> classes, which means the target should contain values in <code>[0, 767]</code>. Every other index will create an out-of-bounds error.</p>