Building a Stack Overflow Tag Labeler
By: Russell Jurney
Tags: Multilabel Classifier, Imbalanced Data, Deep Learning, Classification, Big Data

Imbalanced Labels, Big Data, Extreme Multi-Label Learning

What follows below after an introduction is an email from September, 2019 that details the difficulties I ran into when trying to create a Stack Overflow tag labeler for 10,000+ tags. I hope the “next guy” doing something similar benefits from my example, which is mostly to say, “XML is a hard problem.” No, not that XML and if you think, “he ran into trouble with a common basic example?” then you’re like me: you failed the exam on Extreme Multi-label Learning (XML) with highly imbalanced labels and large datasets. My hope is that by reading the following… you will understand a little bit more about how to scale this common example to be actually useful. XML is a very common AI design pattern!

Extreme Multi-Label Classification

I first read those words in Deep Extreme Multi-label Learning (Zhang, Yan et al). They use the example of a Wikipedia category labeler, very similar to a Stack Overflow tag labeler.

The eXtreme Multi-label Learning (XML) addresses the problem of learning a classifier which can automatically tag a data sample with the most relevant subset of labels from a large label set. For instance, there are more than a million labels (i.e., categories) on Wikipedia and it is expect to build a classifier that can annotate a new article or a web page with a subset of relevant Wikipedia categories. ... However, XML becomes significantly challenging in order to simultaneously deal with massive labels, dimensions and training samples.

As the authors point out… this is different from normal document labeling, this is way harder. The paper referred to in this email is On the Stratification of Multi-label Data (Sechidis, Tsoumakas, Vlahavas), which I started to implement but came to a solution before I finished, so it ended up being more of a guide to a hack. Anyway, later in the email, I lay out how I got past my initial troubles to getting 2,000 labels to work reasonably well and my plan to extend this all the way to 10,000 labels. I hope this will be helpful to someone else!

I’ve put my own edits and comments from today in [square brackets]. I’ve made minor edits for coherence and thanks to Grammarly.

— — — — — — — — — — — — — — — — — — — — — — — —

[Dear Sirs], I wanted to get back to you about what I’m working on. A summary follows. Thank you for taking the time, I really appreciate it.

I see weakly supervised learning and weak supervision (along with Snorkel) as a growth area because neural networks are so data-hungry and labeling data is often prohibitively expensive. Accordingly, I’m writing a book, Weakly Supervised Learning [I got covid and am no longer doing this] that takes the Stack Exchange data dump and applies techniques from NLP along with strategies such as semi-supervised learning, transfer learning, and weak and distant supervision to create real world applications using deep learning.

I set out to create an example model, a Stack Overflow question tagger, by implementing a Kim-CNN classifier. It is a good baseline algorithm for document classification, despite being developed for sentences (questions are not very long). I looked at some examples and created a model to classify the 100 most frequent tags. However… the model wouldn’t learn on any more than the top 10 or so frequent tags. Instead, it predicted the same values for every test record [a sign it was only learning P(label) ]. Class and/or sample weights helped it to learn to predict different values at least, but it had terrible performance — it hadn’t learned anything useful. The class imbalance was high, it turned out and the data used in previous Stack Overflow tagger examples on the web had been balanced off-screen [I wish they had said so]. The imbalance was something like 10,000:1 or more in some cases, ranging from millions to dozens [common to XML, as stated above].

About the imbalance… I looked into balancing multi-label data and came across your paper. In my search I was surprised, multi-label seems to be an oddball area and somewhat neglected — which confuses me because I find most classification problems I encounter are multi-label. I couldn’t find a library to do multi-label balancing for such a large dataset (there are 3.8 million stackoverflow.com questions with answers and at least 1 vote). What I’ve implemented isn’t your full paper with the modifications I mentioned, but is really just up-sampling of each label column to ensure a minimum number of examples for each class [their algorithm only samples each record once, and is thus way smarter than me], and imbalance still accumulates because frequent tags are mixed in with each label’s sub-sample. It still results in a lot of imbalance but at least now the model can learn and perform reasonably well.

The PySpark code that does the stratified over-sampling is here: ch02/get_questions.spark.py. I have thought of implementing your algorithm by adapting it as we discussed to balance the labels by setting the desired proportions of labels to 1 and then under/over-sampling. I am not precisely clear on one step of the model’s implementation, however.

In the Spark script I define three limits:

a stratified sample limit (how many examples at minimum to collect of each class)

a tag limit (the minimum tag frequency, a cutoff point for inclusion of a tag in the output data, below this limit we don’t include it in the training data for the model)

a bad tag limit (‘bad tags’ are between the tag limit and the bad tag limit, these will be enriched/augmented via weak/distant supervision to create more examples. Tags below the bad tag limit are discarded).

Jumping ahead to show you the problem I was working on, the current, working Jupyter Notebook for the classifier model I arrived at is viewable here: Weakly Supervised Learning: Stack Overflow Tag Labeler. It performs reasonably well, using class weights, now that the data is better balanced. You can see the predicted tags versus the training data at the end of the notebook. It works much better on frequent labels than infrequent, but it works ok.