Multi Label Classification in pytorch

SpandanMadan: <p>Hi Everyone,</p>
<p>I’m trying to use pytorch for a multilabel classification, has anyone done this yet?</p>
<p>I have a total of 505 target labels, and samples have multiple labels (varying number per sample).</p>
<p>I tried to solve this by banalizing my labels by making the output for each sample a 505 length vector with 1 at position i, if it maps to label i, and 0 if it doesn’t map to label i.</p>
<p>Then, I tried to use the multilabelmarginloss().</p>
<p>Problem I’m facing - with more iterations, the output of the model should move towards a sparse vector with most elements 0 and a few elements 1, but it is actually moving towards a vector with often very large negative values.</p>
<p>Can some one tell me how to go about this?</p>
<p>Thanks!</p>

smth: <p>Do you have a smaller dataset?</p>
<p>505 target labels with very few samples might just not train well. Maybe add weight decay?</p>

SpandanMadan: <p>It trained well, and gave great results! Thanks for the reply, it was a super silly mistake because I was using the loss function the wrong way. Figured it out after spending some time on the docs <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thank you!</p>

jlee: <p>Can you tell us what kind of loss function you used?</p>

AjayTalati: <p>Hi <a class="mention" href="/u/spandanmadan">@SpandanMadan</a>, great to know you managed to get multi-label classification working in PyTorch, <img src="//discuss.pytorch.org/images/emoji/apple/sunny.png?v=5" title=":sunny:" class="emoji" alt=":sunny:"></p>
<p>I’d like to try doing the same. Can you suggest any small multi-label datasets, (i.e. samples with multiple labels), to start experimenting with?</p>
<p>I’m a novice with this type of problem, so not sure where’s the simplest place to start?</p>
<p>Best regards,</p>
<p>Ajay</p>

dablyo: <p>ok, <a class="mention" href="/u/ajaytalati">@AjayTalati</a>, you can try license number plate dataset.<br>
there are six or seven digits/letters in plate, if choose 7,<br>
every character has 36 possible classes(A-Z,0-9)<br>
so every number plate has  7<em>36 labels as targets, the value 1 indicate the position related to a special character’s value,i</em>36+k(0&lt;=i&lt;=num_character, 0&lt;=k&lt;=35), i indicate the position, and k indicate the value of character.<br>
for example,if target[49]=1, means 1*36+13, the 2nd charater is  ‘M’</p>
<p>i’m also learning pytorch, and take it as an exercise,</p>
<p>the input is BCHW, using multilabelmarginloss()</p>

AjayTalati: <p>Hey <a class="mention" href="/u/dablyo">@dablyo</a> ,</p>
<p>thanks a lot, great idea, <img src="//discuss.pytorch.org/images/emoji/apple/smile.png?v=5" title=":smile:" class="emoji" alt=":smile:"> ! And thank you for the example too!</p>
<p>Have you got a link to the data set you are using please? I’d like to work on it too as an exercise <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:">,</p>
<p>I’ve tried a few experiments with using <code>multilabelmarginloss()</code>, but I couldn’t get it to work ?</p>
<p>I could still train multiple, multi-class, classifiers on the same dataset. So for example I could train, 7 classifiers, one for each digit/letter in a number plate, but that’s not too smart because there’s some structure in the sequence of digits/letters, at least for UK number plates <img src="//discuss.pytorch.org/images/emoji/apple/smile.png?v=5" title=":smile:" class="emoji" alt=":smile:"> .</p>
<p>Kind regards,</p>
<p>Ajay</p>

dablyo: <p>In <a href="http://matthewearl.github.io/2016/05/06/cnn-anpr/" class="inline-onebox" rel="noopener nofollow ugc">Number plate recognition with Tensorflow - Matt's Ramblings</a>, the author has generate dataset from backgroupd picture and number plate font file, you can learn from it.</p>
<p>I’ve meet some problem<br>
while training ,in 2nd minibatch, the output of multilabelmarginloss() is zero, i cann’t find out the reason.</p>
<p>the inputting image size 224<em>224, the target vector width is 252(7</em>36),<br>
‘X’  versus  000000000000000000000001000000000000<br>
‘A’  versus  100000000000000000000000000000000000<br>
the source code is as follows:</p>
<blockquote>
<p>import …</p>
</blockquote>
<blockquote>
<p>DIGITS = “0123456789”<br>
LETTERS = “ABCDEFGHIJKLMNOPQRSTUVWXYZ”<br>
CHARS = LETTERS + DIGITS<br>
NPLEN=7<br>
NUM_CLASSES=252</p>
</blockquote>
<blockquote>
<p>class anprmodel(nn.Module):</p>
</blockquote>
<blockquote>
<pre><code>def __init__(self):
    super(anprmodel,self).__init__()
    self.num_classes=NUM_CLASSES
    self.conv1=nn.Conv2d(1,48,kernel_size=5,padding=2)  
    self.pool1=nn.MaxPool2d(kernel_size=(2,2),stride=2)
    self.conv2=nn.Conv2d(48,64,kernel_size=5,padding=2)
    self.pool2=nn.MaxPool2d(kernel_size=(2,2),stride=(2,2))
    self.conv3=nn.Conv2d(64,128,kernel_size=5,padding=2)
    self.pool3=nn.MaxPool2d(kernel_size=(2,2),stride=(2,2))        
    self.fc1=nn.Linear(28*28*128,2048)  
    self.fc2=nn.Linear(2048,NUM_CLASSES)
    
def forward(self,x): 
    x=F.relu(self.pool1(self.conv1(x)))  #input: 224*224
    x=F.relu(self.pool2(self.conv2(x)))  #input 112*112
    x=F.relu(self.pool3(self.conv3(x)))  #input 56*56
    x=x.view(-1,28*28*128)             #28*28
    x=F.relu(self.fc1(x))
    x=self.fc2(x)                            #output: 252
    return x
</code></pre>
</blockquote>
<blockquote>
<p>class NPSET(torch_utils_data.Dataset):<br>
picroot=‘np’</p>
<pre><code>def code_to_vec(self,p, code):
    def char_to_vec(c):
        y = np.zeros((len(CHARS),))
        y[CHARS.index(c)] = 1.0
        return y
    c = np.vstack([char_to_vec(c) for c in code])
    return c.flatten()
</code></pre>
</blockquote>
<blockquote>
<pre><code>def __getitem__(self,index):
    label,img=self.labels[index], self.dataset[index]
    if self.data_transform is not None:
        img=self.data_transform(img)
    labelarray=self.code_to_vec(1,label)
    return (img,labelarray)
</code></pre>
</blockquote>
<blockquote>
<pre><code>def __len__(self):
    return self.len
</code></pre>
</blockquote>
<blockquote>
<pre><code>def __init__(self,root,data_transform=None):
    self.picroot=root
    self.data_transform=data_transform
</code></pre>
</blockquote>
<blockquote>
<pre><code>    if not os.path.exists(self.picroot):
        raise RuntimeError('{} doesnot exists'.format(self.picroot))
    for root,dnames,filenames in os.walk(self.picroot):
        imgs=[] 
        labels=[]
        for filename in filenames:
            picfilename=os.path.join(self.picroot,filename)  #file name:
            im=cv2.imread(picfilename,cv2.IMREAD_GRAYSCALE)
            im=cv2.resize(im,(224,224))
            imgs.append(im)
            m=filename.split('_')  #filename style: xxxxxxxx_yyyyyyy_z.png
            labels.append(m[1])
        self.dataset=imgs
        self.labels=labels
        self.len=len(filenames)
</code></pre>
<p>def accuracy(output,target):  <span class="hashtag-raw">#Tensor:Tensor</span>  <span class="hashtag-raw">#size:</span> batchsize<em>252<br>
batchsize=output.size(0)<br>
assert(batchsize==target.size(0))<br>
p=torch.chunk(output,7,1)    <span class="hashtag-raw">#p</span>[0]–p[6], batchsize</em>36<br>
t=torch.chunk(target,7,1)</p>
<pre><code>a=np.ones((batchsize,1),np.dtype('i8'))*7      #7,7,7,7,7.....7   num is batchsize
ts=torch.from_numpy(a)   #LongTensor, tmp, and will be cut
ps=torch.from_numpy(a)

for i in range(0,NPLEN):   # the index of max value in every segment
    _,pred=torch.max(p[i],1)
    ps=torch.cat((ps,pred),1)
    _,pred=torch.max(t[i],1)
    ts=torch.cat((ts,pred),1)
sub=torch.LongTensor([1,2,3,4,5,6,7])    
ts=torch.index_select(ts,1,sub)   #LongTensor
ps=torch.index_select(ps,1,sub) #LongTensor
tspseq=torch.eq(ts,ps)      #ByteTensor
tspseqsum=torch.sum(tspseq,1)   #ByteTensor ,it will be 7 if right
a=np.ones((batchsize,1),np.uint8)*7   #byte ndarray
result=torch.eq(tspseqsum,torch.from_numpy(a))
return batchsize,torch.sum(result)   #batchsize  number of right
</code></pre>
<p>class recMeter(object):<br>
def <strong>init</strong>(self):<br>
self.reset()<br>
is_best=False<br>
best=0<br>
current=0</p>
<pre><code>def reset(self):
    self.right = 0
    self.sum = 0
    current=0

def updatecnt(self, n, r):
    self.right+=r
    self.sum+=n
    
def updateaccurate(self):
    self.current=self.right/self.sum
    if ac &gt; best:
        is_best=True
        best=ac
</code></pre>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
model=anprmodel()<br>
model.cuda()<br>
cudnn.benchmark=True<br>
batch_size=10<br>
data_transform=transforms.Compose([transforms.ToTensor(),<br>
transforms.Normalize((107.897212036,), (3893.57887653,)),<br>
])<br>
npset = NPSET(root=‘/home/wang/git/nppic/nproot/plate’, data_transform=data_transform)<br>
nploader = torch.utils.data.DataLoader(npset, batch_size=batch_size, shuffle=True, num_workers=1)  <span class="hashtag-raw">#train</span><br>
npvalset=NPSET(root=‘/home/wang/git/nppic/npval/plate’, data_transform=data_transform)<br>
npvalloader=torch.utils.data.DataLoader(npvalset, batch_size=batch_size, shuffle=False, num_workers=1) <span class="hashtag-raw">#validate</span><br>
criterion=nn.MultiLabelMarginLoss()<br>
optimizer=torch.optim.SGD(model.parameters(),0.1,momentum=0.9)</p>
<pre><code>meter=recMeter()
for epoch in range(0,1):
    #Sets the learning rate to the initial LR decayed by 10 every 30 epochs
    lr=0.1*(0.1**(epoch//30))
    #for param_group in optimizer.param_groups:
    #    param_group['lr']=lr
    #train
    model.train()
    for i,data in enumerate(nploader):
        inputs,targets = data   #inputs size: batchsize*224*224
        inputs=torch.unsqueeze(inputs,1)  ##inputs size: batchsize*1*224*224
        targets=torch.LongTensor(np.array(targets.numpy(),np.long))
        targets=targets.cuda()
        inputs=inputs.cuda()
        input_var=torch.autograd.Variable(inputs)
        target_var=torch.autograd.Variable(targets)
        
        optimizer.zero_grad()
        output_var=model(input_var)
        #porcess loss
        character_loss=criterion(output_var,target_var)
    
        # compute gradient and do SGD step
        character_loss.backward()
        optimizer.step()
</code></pre>
</blockquote>
<p><strong>I’ve execute train loop in python console by this way:</strong></p>
<blockquote>
<p>npiter=iter(nploader)</p>
</blockquote>
<p><strong>then</strong></p>
<blockquote>
<p>(inputs,targets)=npiter.next()<br>
inputs=torch.unsqueeze(inputs,1)<br>
targets=torch.LongTensor(np.array(targets.numpy(),np.long))<br>
targets=targets.cuda()<br>
inputs=inputs.cuda()<br>
input_var=torch.autograd.Variable(inputs)<br>
target_var=torch.autograd.Variable(targets)<br>
optimizer.zero_grad()<br>
output_var=model(input_var)<br>
character_loss=criterion(output_var,target_var)<br>
character_loss.backward()<br>
optimizer.step()<br>
print(‘Loss: {:.6f}’.format(character_loss.data[0]))</p>
</blockquote>
<p><strong>from 2nd mini batch, the loss become 0.</strong><br>
<strong>I’ve stucked here.</strong></p>

sshuair: <p>I got simiar result, I got zero loss from the second epoch. Do you find the reason?</p>

AjayTalati: <p>Hi there,</p>
<p>I can’t get multi-label classification working either, but <a class="mention" href="/u/bartolsthoorn">@bartolsthoorn</a> and <a class="mention" href="/u/mratsim">@mratsim</a> have found possible ways to do it here</p>
<aside class="quote" data-post="2" data-topic="2303">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bartolsthoorn/48/1182_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/calculating-accuracy-for-a-multi-label-classification-problem/2303/2">Calculating accuracy for a multi-label classification problem</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I came up with the answer: torch.sigmoid(classified_labels).data &gt; 0.5 will give the correct labels with MultiLabelSoftMarginLoss().
  </blockquote>
</aside>

<p>hope that helps?</p>
<p>Aj</p>

bartolsthoorn: <p><a class="mention" href="/u/ajaytalati">@AjayTalati</a> OK I wrote a simple example here: <a href="https://gist.github.com/bartolsthoorn/36c813a4becec1b260392f5353c8b7cc" rel="nofollow noopener">https://gist.github.com/bartolsthoorn/36c813a4becec1b260392f5353c8b7cc</a></p>
<p>For accuracy it is important to note that you can pass the output first through nn.Sigmoid and everything &gt; 0.5 is true (look at the Sigmoid function: <a href="https://en.wikipedia.org/wiki/Sigmoid_function" rel="nofollow noopener">https://en.wikipedia.org/wiki/Sigmoid_function</a></p>

AjayTalati: <p>Hey <a class="mention" href="/u/bartolsthoorn">@bartolsthoorn</a>,</p>
<p>that’s really helpful, thank you very much <img src="//discuss.pytorch.org/images/emoji/apple/smile.png?v=5" title=":smile:" class="emoji" alt=":smile:">, really nice example <img src="//discuss.pytorch.org/images/emoji/apple/smile.png?v=5" title=":smile:" class="emoji" alt=":smile:"></p>
<p>Kind regards,</p>
<p>Ajay</p>

mratsim: <p><a class="mention" href="/u/ajaytalati">@AjayTalati</a></p>
<p>Either after your last fc you do a sigmoid and then you use <code>BCELoss</code> or <code>F.binary_crossentropy</code> as your criterion/lossfunction</p>
<p>Or you directly use <code>MultiLabelSoftMarginLoss</code> as your loss function (it comes with sigmoid inside)</p>
<p>Now once you have your prediction, you need to threshold. 0.5 is the default naive way but it’s probably not optimal. In any case, once you get there, great !</p>
<h4>Next part is technical optimization, you can do Multilabel classification without</h4>
<p>Regarding the threshold, you might want to optimize either a common threshold for all your outputs (it can be 0.2, 0.5, 0.123456 who knows) or optimize a threshold per label class, especially if your classes as unbalanced.<br>
You will need a solid validation set and a MultiLabel evaluation metrics (Hamming Loss, F1-score, Fbeta score).</p>
<p>An example code for the first strategy is <a href="https://www.kaggle.com/paulorzp/planet-understanding-the-amazon-from-space/find-best-f2-score-threshold" rel="nofollow noopener">here on Kaggle</a>.</p>
<p>For the second strategy, I’m deep into various papers myself so I can’t help yet.<br>
One thing to keep in mind is your “best threshold” will probably overfit the validation set, so use regularization, cross-validation or other anti-overfitting strategy.</p>

nabergh: <p>Does anyone understand how MultiMarginLoss is calculated exactly? I’m not sure I understand completely.</p>
<blockquote>
<p><code>loss(x, y) = sum_ij(max(0, 1 - (x[y[j]] - x[i]))) / x.size(0)</code><br>
where i == 0 to x.size(0), j == 0 to y.size(0), y[j] != 0, and i != y[j] for all i and j.</p>
</blockquote>
<p>The docs say y is a set of indices. If <code>y[j] != 0</code> is enforced, how do you check the loss for class 0? Also if x belongs to two  or more classes, how does <code>max(0, 1 - (x[y[j]] - x[i]))</code> contribute to the loss when both <code>y[j]</code> and <code>x[i]</code> are classes that x belongs to?</p>
<p>I also don’t know how to find the source code for MultiMarginLoss the <a href="http://pytorch.org/docs/_modules/torch/nn/modules/loss.html#MultiLabelMarginLoss" rel="noopener nofollow ugc">docs link</a> isn’t very informative.</p>

mratsim: <p>Had the same issue with the loss documentation. Myguess is that the current code is probably this one <a href="https://github.com/pytorch/pytorch/blob/master/torch/legacy/nn/MultiLabelMarginCriterion.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/legacy/nn/MultiLabelMarginCriterion.py</a> and what is in the docs a stub before it’s converted to the new APIs</p>

ahkarami: <p>Hi Everyone,</p>
<p>I’m trying to Finetune the pre-trained convnets (<em>e.g.</em>, resnet50) for a data set, which have 3 categories. In fact, I want to extend the introduced code of ‘<em><strong>Transfer Learning tutorial</strong></em>’ (<a href="http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#sphx-glr-beginner-transfer-learning-tutorial-py" rel="noopener nofollow ugc">Transfer Learning tutorial</a>) for a new data set which have <strong>3</strong> categories. In addition, in my data set each image has just one label (<em>i.e.</em>, each train/val/test image has just one label). Could you help me please to do that?<br>
I have changed the above-mentioned code as follows:</p>
<ol>
<li>I have changed the parameters of <code>nn.Linear</code> as follow:</li>
</ol>
<blockquote>
<p>num_ftrs = model_conv.fc.in_features<br>
model_conv.fc = nn.Linear(num_ftrs, 3) # 3 means we have 3 class labels</p>
</blockquote>
<ol start="2">
<li>
<p>I have changed the Loss function:<br>
<code>criterion = nn.NLLLoss()</code></p>
</li>
<li>
<p>I have changed the ‘<strong>train_model</strong>’ method as follow:</p>
</li>
</ol>
<blockquote>
<p>…<br>
m = nn.LogSoftmax()<br>
outputs = model(inputs)<br>
_, preds = torch.max(outputs.data, 1)<br>
loss = criterion(m(outputs), labels)</p>
</blockquote>
<p>However, my obtained result isn’t good at all. As a result, my precise questions are as follows:</p>
<ol>
<li>In these cases which Loss function must be used?</li>
<li>Are those changes for training the model and compute the loss correct?</li>
</ol>

mratsim: <p><a class="mention" href="/u/ahkarami">@ahkarami</a> I think you should create a separate topic for your issue which is very different from the original post. You are doing Multiclass classification instead of multilabel.</p>
<p>Your loss function is correct btw.</p>

ahkarami: <p>Thank you very much for your help. I agree with you there. As a result, I will create a new topic (<strong><em>Multiclass Classification in PyTorch</em></strong>).</p>

dablyo: <p><a class="mention" href="/u/ajaytalati">@AjayTalati</a><br>
<a class="mention" href="/u/mratsim">@mratsim</a></p>
<p>I’ve used MultiLabelSoftMarginLoss and Adam optimizer，the loss looked well.<br>
the SGD optimizer worked properly also, and same as last fc along with sigmoid,then BCELoss.</p>
<p>the MultiLabelMarginLoss doesn’t work, loss become 0 in 2nd minibatch.</p>
<p>the last loss is 0.08…, cann’t  become smaller.<br>
Train Epoch: 29 (19%)Loss: 0.081794<br>
Train Epoch: 29 (39%)Loss: 0.080127<br>
Train Epoch: 29 (59%)Loss: 0.083426<br>
Train Epoch: 29 (79%)Loss: 0.086233<br>
Train Epoch: 29 (99%)Loss: 0.082037</p>
<p>but for 252(36*7) destinations, for different image, for example 10 images from test set, the model gave a same prediction,just like:<br>
0.0006  0.0006  0.0006  …   0.0891  0.0742  0.1139<br>
0.0006  0.0006  0.0006  …   0.0891  0.0742  0.1139<br>
0.0006  0.0006  0.0006  …   0.0891  0.0742  0.1139<br>
…             ⋱             …<br>
0.0006  0.0006  0.0006  …   0.0891  0.0742  0.1139<br>
0.0006  0.0006  0.0006  …   0.0891  0.0742  0.1139<br>
0.0006  0.0006  0.0006  …   0.0891  0.0742  0.1139<br>
[torch.FloatTensor of size 10x252]</p>
<p>I’m whole confused</p>

SpandanMadan: <p>Hi,</p>
<p>I used BCELoss. It’s the standard for multi label classification in many ways. Give it a shot.</p>