Weakly Supervised Learning: Classification with limited annotation capacity | by Ved Vasu Sharma | Towards Data Science

With the introduction of a wide range of pre-trained models trained on humongous data, the success of Machine learning in industrial applications has grown exponentially. While we can easily utilize these models by transfer learning either as feature extractors or by fine-tuning on our specific dataset, but generally the performance isn’t great if the pattern is significantly different from the data on which these models are originally trained.
Especially in the case of Natural Language Processing, there are many patterns specific to the data which often reduces the applicability of pre-trained models. At the same time, the challenges involved in getting application-specific annotations not only reduces the usability of the data but also of supervised (transfer) learning for modeling.
Since unannotated text data is generally available in abundance, unsupervised/ weakly supervised approaches become the first-hand choice for data exploration and even for modeling. Because building a team of annotators with the domain expertise is not only expensive but also time-consuming, recently, practitioners have been turning to the weaker forms of supervision which is the main focus of this blog.
As mentioned in Wikipedia, Weak supervision is a branch of machine learning where noisy, limited, or imprecise sources are used to provide supervision signal for labeling large amounts of training data in a supervised learning setting. This approach alleviates the burden of obtaining hand-labeled data sets, which can be costly or impractical. Instead, inexpensive weak labels are employed with the understanding that they are imperfect, but can nonetheless be used to create a strong predictive model.
For a large scale dataset such as Amazon Reviews for Sentiment, the aim is to identify broad categories regarding what users are mentioning in the negative reviews for books and further build a predicted model which can be used to provide categorical feedback to the sellers.
Heuristically extract reviews with low rating
Heuristically extract reviews with low rating
import pandas as pd# Load datasetdf = pd.read_csv("amazon_product_reviews.csv")# Negative Reviews (With rating 1-2)df = df[df['Rating'] == '1-2']# Sample set of negative Review that mentions bookdf = df[df['Amazon Review'].str.contains(    'book|publication|novel|paperback|hardcover')].sample(50000)
2. Standard preprocessing of text data
import reimport nltkimport itertoolsimport contractions# Load datasetdf = pd.read_csv("amazon_product_reviews.csv")column = 'Amazon Review'# Lowercase textdf[column] = df[column].apply(    lambda x: x.lower())# Fix contraction in the wordsdf[column] = df[column].apply(    lambda x: contractions.fix(str(x)))# Standardise wordsdf[column] = df[column].apply(    lambda x: ''.join(''.join(c)[:2] for _, c in itertools.groupby(x)))# Remove punctuationspunctuations_regex="!#%&\'()\*+,-/:;<=>?@\[\]^_`{|}~.1234567890"df[column] = df[column].apply(    lambda x: re.sub('[%s]'%re.escape(punctuations_regex), ' ', x))# Tokenize wordsdf[column] = df[column].apply(    lambda x: ' '.join(nltk.word_tokenize(x)))
3. Feature Extraction and Clustering
ngram_range and min_df can be experimented further to optimize for more coverage of important phrases and less of unimportant phrases.
n_clusters is optimized based on Elbow methods + the availability of experts for annotating clusters in later steps.
from sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.cluster import KMeans# Extract TFIDF features with 1-4 gramsTFIDF_PARAMS = {    'strip_accents': 'ascii',    'stop_words': 'english',    'sublinear_tf': True,    'ngram_range': (1, 4),    'min_df': 0.003,}vectorizer = TfidfVectorizer(**TFIDF_PARAMS)tfidf_model = vectorizer.fit(df[column])train_features = tfidf_model.transform(df[column])# Identify clusters on the modelcluster_model = KMeans(n_clusters=200).fit(train_features)df['cluster_label'] = cluster_model.labels_
4. Analysing and annotating clusters
All the clusters are labeled by domain experts and the clusters label is assigned to all cluster samples. This immensely reduces the annotation time and cost as only n clusters are labeled instead of all 3.6 million samples.
Finally, the sampling results for the “Final Category” on the extracted dataset showed that it's 85 (±5) % accurate.
Finally, a large training dataset is curated by running a clustering model on the whole unstructured data, and the labels are assigned based on clusters identifier. The final predictive model is trained on this dataset in a supervised fashion which not only deals with errors in the dataset but also provides high-quality predictions at higher confidence (based on confidence threshold).
# Training XGBoost (Weak supervision)from xgboost import XGBClassifierfrom sklearn.metrics import classification_reportclf = XGBClassifier(n_jobs= -1, objective='multi:softmax')clf = clf.fit(X_train, y_train)scores = clf.predict_proba(X_test)y_true = pd.DataFrame(scores,columns=clf.classes_).idxmax(axis=1)print (classification_report(y_test, y_true, digits=4))
The model performed exceptionally well under Weak Supervision but the accuracy can drop from 5–15% in live settings depending on the category and purity of clusters belonging to that category.
This approach definitely helps us kickstart with the problems without spending a significant amount of time and money on building large training data. Furthermore, it can be an iterative process of learning algorithms based on heuristics, hierarchical clustering, and modeling until we obtain high-quality ground truth labels.
The performance of the predictive model improves over time by retraining on a better quality dataset gathered by the recurring sampling of model predictions by domain experts.