Multi-label classification as array output in pytorch

sr24: <p>Hi, I’m trying to make a classifier using a CNN. The data points consist of a 70x70 image and 5 labels represented as a list of length 5 as each image contains up to 5 digits.</p>
<p>Here’s example of a label for an image which contains the digits 1,5 and 9.</p>
<pre><code class="lang-auto">[1, 5, 9, -1, -1]
</code></pre>
<p>Below are snippets of parts of my code so far, for reference…</p>
<pre><code class="lang-auto">class complexNet(nn.Module):
    def __init__(self):
        super(complexNet, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, ...)
        ...
        self.fc5 = nn.Linear(120, 11) # 11 possible classes: digits 0-9 and no digit, -1.

    def forward(self, x):
        x = x.cuda()
        x = self.pool(torch.sigmoid(self.conv1(x)))
        ...   
        x = x.view(-1, c*w*h )
        ...
        x = self.fc5(x)
        return x

model = complexNet()
optimizer = optim.Adam(...)
loss = nn.MultiLabelSoftMarginLoss()

for i, batch in enumerate(train_loader, 0):
        x,y = batch
        logit = model(x)
        _, predicted = torch.max(logit, dim=1)
        print(predicted)
        J = loss(logit,y)
        print(J.item())
</code></pre>
<p>The print statement outputs something like this, where each tensor is of length 10 since I’m using a batch size of 10:</p>
<pre><code class="lang-auto">...
tensor([6, 6, 6, 6, 6, 6, 6, 6, 6, 6], device='cuda:0')
tensor([ 6,  6,  6,  6,  6,  6,  6,  6, 10,  6], device='cuda:0')
tensor([6, 4, 6, 5, 6, 6, 6, 5, 6, 6], device='cuda:0')
tensor([6, 6, 6, 6, 6, 6, 5, 6, 6, 6], device='cuda:0')
...
</code></pre>
<p>Ideally, an example of a predicted value would look like the following:</p>
<pre><code class="lang-auto">tensor([
 [1,2,3,4,5],
 [1,0,0,-1,-1],
 [5,2,3,0,-1],
 [9,2,3,3,-1],
 [0,-1,-1,-1,-1]
...
...
], device='cuda:0')
</code></pre>
<p>In addition, unsurprisingly the loss function yields the following error:</p>
<pre><code class="lang-auto">D:\SoftwarePackages\Anaconda\lib\site-packages\torch\nn\functional.py:1594: UserWarning: Using a target size (torch.Size([10, 5])) that is different to the input size (torch.Size([10, 11])) is deprecated. Please ensure they have the same size.
  "Please ensure they have the same size.".format(target.size(), input.size()))
...
ValueError: Target and input must have the same number of elements. target nelement (50) != input nelement (110)
</code></pre>
<p>For simplicity, we can break the problem down into 2 main concerns for the time being:</p>
<ol>
<li>I would like each number in the tensor to instead be an array of length 5, where each number is a digit from the image.</li>
<li>Should this be done in the forward function, or within the training loop?</li>
</ol>
<p>I’ve mostly tried to solve this problem based off of advice from 2 other topics, but have not had any success.</p><aside class="quote" data-post="13" data-topic="10440">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/a-model-with-multiple-outputs/10440/13">A model with multiple outputs</a> 
  </div>
  <blockquote>
    Sure, that should also work! 
Let me know, if you encounter any problems. <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title="wink" alt="wink" class="emoji">
  </blockquote>
</aside>
<aside class="quote quote-modified" data-post="1" data-topic="32203">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vinill/48/8164_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-kind-of-loss-is-better-to-use-in-multilabel-classification/32203">What kind of loss is better to use in multilabel classification?</a> 
  </div>
  <blockquote>
    I am currently working on my mini-project, where I predict movie genres based on their posters. So in the dataset that I have, each movie can have from 1 to 3 genres, therefore each instance can belong to multiple classes. I have total of 15 classes(15 genres). I use mini-batch of 4.When I train my classifier, my labels is a list of 3 elements and it looks like that: 
tensor([[ 2., 10.,  5.], 
[ 2.,  5.,  0.], 
[14.,  0.,  0.], 
[ 1.,  0.,  0.]]) , where 0 means there is no genre for that positi…
  </blockquote>
</aside>

<p>Any help at all is hugely appreciated, Thank you!</p>

shartzog: <p>What does the -1 mean? What’s the significance of the ordinal of each target int (which I assume is a class designation)? What’s preventing you from using a more “traditional” approach, e.g. a standard multi-label classifier? You’d definitely have more than 5 labels given your problem description, but it’d be much more straightforward to implement and would still let you rank your labels according to the strength of the match. Are you familiar with expressing labels using multi-hot encoding? If not, I can give you a rundown. If so, I probably misunderstood your question and don’t know enough to help… <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

sr24: <p>Hi Sam, thank you for responding to my post,</p>
<p>I guess it was pretty important and I should have clarified it in my post, but the digits on the image are to be displayed from left to right in the array, as they appear in the same order in the image.<br>
For example, if the the left most digit in the image is 1, then 5 appears somewhere to the right and 9 is the right-most digit, then the label is [1,5,9,-1,-1]. We include the two -1’s in the array as placeholder values, since the list must be of length 5 as that is how the labels are represented in the original dataset. I figured keeping all lists at length 5 would simplify training, but maybe I’m wrong? Hopefully this clears up the problem a little bit.</p>
<p>The reason I chose to use a CNN was because we are predicting many mutually non-exclusive class labels corresponding with a 70x70x1 image input, so I figured a convolutions was the best approach to analyzing visual imagery. But of course I’m open to any and all suggestions/advice! I’m not very familiar with expressing labels using multi-hot encoding, could you could provide a quick head-start on the topic and how it could be applied in the context of this problem?</p>

shartzog: <p>I’m still struggling to understand your use case, so I’m not sure this will help, but to multi-hot encode a class list dataset with integer labels of:<br>
Sample 0: [3, 7, 4]<br>
Sample 1: [2, 4]<br>
Sample 2: [9, 1, 6, 2]<br>
Sample 3: [8, 9]<br>
you’d transform it into something like:</p>
<pre><code class="lang-auto">Label:     0  1  2  3  4  5  6  7  8  9  ...  N
           |  |  |  |  |  |  |  |  |  |       |
           v  v  v  v  v  v  v  v  v  v       v
Sample 0: [0, 0, 0, 1, 1, 0, 0, 1, 0, 0, ..., 0]
Sample 1: [0, 0, 1, 0, 1, 0, 0, 0, 0, 0, ..., 0]
Sample 2: [0, 1, 1, 0, 0, 0, 1, 0, 0, 1, ..., 0]
Sample 3: [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, ..., 0]
</code></pre>
<p>where N is the total number of classes.  Once you’ve transformed the labels, it’s pretty straightforward to put together a simple CNN using <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">BCELossWithLogits</a>.  This is a common strategy for basic CNNs and is the basis of most “multi-label CNN” tutorials, but you’ll notice it does NOT preserve order or provide bounding boxes for detected entities/labeled objects.  If you’re looking for that more advanced functionality, I’d recommend looking at:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/AlexeyAB/darknet">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/AlexeyAB/darknet" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/c63a15079c8a1ab11cc935806c43caae528fbb988b41ad9e2906be4b88b33041/AlexeyAB/darknet" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/AlexeyAB/darknet" target="_blank" rel="noopener nofollow ugc">GitHub - AlexeyAB/darknet: YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks...</a></h3>

  <p>YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet ) - GitHub - AlexeyAB/darknet: YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object ...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
but that would take you out of the torch framework altogether.  Hopefully, someone more knowledgeable can comment and point you in the right direction for using torch.  If not, here’s a link to a post that discusses multi-label classification problems in general.  You might find  some inspiration there…</p><aside class="quote" data-post="45" data-topic="905">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/multi-label-classification-in-pytorch/905/45">Multi Label Classification in pytorch</a> 
  </div>
  <blockquote>
    Here is a very simple dummy example: 
model = nn.Linear(20, 5) # predict logits for 5 classes
x = torch.randn(1, 20)
y = torch.tensor([[1., 0., 1., 0., 0.]]) # get classA and classC as active

criterion = nn.BCEWithLogitsLoss()
optimizer = optim.SGD(model.parameters(), lr=1e-1)

for epoch in range(20):
    optimizer.zero_grad()
    output = model(x)
    loss = criterion(output, y)
    loss.backward()
    optimizer.step()
    print('Loss: {:.3f}'.format(loss.item()))
  </blockquote>
</aside>