BERT for Text Classification with NO model training | by Mauro Di Pietro | Towards Data Science

Are you struggling to classify text data because you don’t have a labeled dataset? In this article, using BERT and Python, I will explain how to perform a sort of “unsupervised” text classification based on similarity.
NLP (Natural Language Processing) is the field of artificial intelligence that studies the interactions between computers and human languages, in particular how to program computers to process and analyze large amounts of natural language data. NLP is often applied for classifying text data. Text classification is the problem of assigning categories to text data according to its content. In order to carry out a classification use case, you need a labeled dataset for machine learning models training. So what happens if you don’t have one?
This kind of situation happens in the real world more often than you think. Nowadays, AI is so hyped that firms want to use it even when they don’t have data. In particular, most non-technical people don’t fully get the concept of “target variable” and how it is used in supervised machine learning. So how can you build a classifier when you have text data but no label? In this tutorial, I am going to explain a strategy that applies W2V and BERT to classify text by word vector similarity.
I will present some useful Python code that can be easily applied in other similar cases (just copy, paste, run) and walk through every line of code with comments so that you can replicate this example (link to the full code below).
github.com
I will use the “News category dataset” in which you are provided with news headlines from the year 2012 to 2018 obtained from HuffPost and you are asked to classify them with the right category, therefore this is a multiclass classification problem (link below).
www.kaggle.com
In particular, I will go through:
Setup: import packages, read data.
Preprocessing: clean text data.
Create Target Clusters: use Word2Vec with gensim to build the target variable.
Feature Engineering: Word Embedding with transformers and BERT.
Model Design & Testing: assign observations to clusters by Cosine Similarity and evaluate the performance.
Explainability: understand how the model produces results.
First of all, I need to import the following packages:
## for dataimport jsonimport pandas as pdimport numpy as npfrom sklearn import metrics, manifold## for processingimport reimport nltk## for plottingimport matplotlib.pyplot as pltimport seaborn as sns## for w2vimport gensimimport gensim.downloader as gensim_api## for bertimport transformers
The dataset is contained into a json file, so I will first read it into a list of dictionaries with json and then transform it into a pandas Dataframe.
lst_dics = []with open('data.json', mode='r', errors='ignore') as json_file:    for dic in json_file:        lst_dics.append( json.loads(dic) )## print the first onelst_dics[0]
The original dataset contains over 30 categories, but for the purposes of this tutorial, I will work with a subset of 3: Entertainment, Politics, and Tech.
## create dtfdtf = pd.DataFrame(lst_dics)## filter categoriesdtf = dtf[ dtf["category"].isin(['ENTERTAINMENT','POLITICS','TECH'])        ][["category","headline"]]## rename columnsdtf = dtf.rename(columns={"category":"y", "headline":"text"})## print 5 random rowsdtf.sample(5)
As you can see, the dataset includes a target variable as well. I won’t be using it for modeling, just for performance evaluation.
So we’ve got some raw text data and we are tasked to classify it into the 3 categories (Entertainment, Politics, Tech) we know nothing about. Here’s what I am planning to do:
clean data and embed it into the vector space,
create a topic cluster for each category and embed it into the vector space,
calculate similarities between every text vector and the topic clusters, then assign it to the closest cluster.
That is why I called it “a sort of unsupervised text classification”. It’s a really basic idea, but the execution can be tricky.
Now that’s all set, let’s get started.
The absolute first step is to preprocess the data: cleaning text, removing stop words, and applying lemmatization. I will write a function and apply it to the whole data set.
'''Preprocess a string.:parameter    :param text: string - name of column containing text    :param lst_stopwords: list - list of stopwords to remove    :param flg_stemm: bool - whether stemming is to be applied    :param flg_lemm: bool - whether lemmitisation is to be applied:return    cleaned text'''def utils_preprocess_text(text, flg_stemm=False, flg_lemm=True, lst_stopwords=None):    ## clean (convert to lowercase and remove punctuations and       characters and then strip)    text = re.sub(r'[^\w\s]', '', str(text).lower().strip())                ## Tokenize (convert from string to list)    lst_text = text.split()        ## remove Stopwords    if lst_stopwords is not None:        lst_text = [word for word in lst_text if word not in                     lst_stopwords]                    ## Stemming (remove -ing, -ly, ...)    if flg_stemm == True:        ps = nltk.stem.porter.PorterStemmer()        lst_text = [ps.stem(word) for word in lst_text]                    ## Lemmatisation (convert the word into root word)    if flg_lemm == True:        lem = nltk.stem.wordnet.WordNetLemmatizer()        lst_text = [lem.lemmatize(word) for word in lst_text]                ## back to string from list    text = " ".join(lst_text)    return text
That function removes a set of words from the corpus if given. I can create a list of generic stop words for the English vocabulary with nltk (we could edit this list by adding or removing words).
lst_stopwords = nltk.corpus.stopwords.words("english")lst_stopwords
Now I shall apply the function to the whole dataset and store the result in a new column named “text_clean” that I am going to use as a corpus.
dtf["text_clean"] = dtf["text"].apply(lambda x:           utils_preprocess_text(x, flg_stemm=False, flg_lemm=True,           lst_stopwords=lst_stopwords))dtf.head()
We have our preprocessed corpus, consequently the next step is to build the target variable. Basically, we’re here:
The objective of this section is to create some keywords which can represent the context of each category. By performing some text analysis, you can easily discover that the 3 most frequent words are “movie”, “trump”, and “apple” (for a detailed text analysis tutorial you can check this article). I’d suggest starting with those keywords.
Let’s take the Politics category for instance: the word “trump” can have different meanings, so we need to add keywords to avoid polysemy problems (i.e. “donald”, “republican”, “white house”, “obama”). This task could be carried out manually or you could use the assistance of a pre-trained NLP model. You can load a pre-trained Word Embedding model from genism-data like this:
nlp = gensim_api.load("glove-wiki-gigaword-300")
The gensim package has a very convenient function that returns the most similar words for any given word into the vocabulary.
nlp.most_similar(["obama"], topn=3)
I shall use that to create a dictionary of keywords for each category:
## Function to applydef get_similar_words(lst_words, top, nlp):    lst_out = lst_words    for tupla in nlp.most_similar(lst_words, topn=top):        lst_out.append(tupla[0])    return list(set(lst_out))## Create Dictionary {category:[keywords]}dic_clusters = {}dic_clusters["ENTERTAINMENT"] = get_similar_words(['celebrity','cinema','movie','music'],                   top=30, nlp=nlp)dic_clusters["POLITICS"] = get_similar_words(['gop','clinton','president','obama','republican']                  , top=30, nlp=nlp)dic_clusters["TECH"] = get_similar_words(['amazon','android','app','apple','facebook',                   'google','tech'],                    top=30, nlp=nlp)## print somefor k,v in dic_clusters.items():    print(k, ": ", v[0:5], "...", len(v))
Let’s try to visualize those keywords in a 2D space by applying a dimensionality reduction algorithm (i.e. TSNE). We want to make sure that the clusters are well separated from each other.
## word embeddingtot_words = [word for v in dic_clusters.values() for word in v]X = nlp[tot_words]        ## pcapca = manifold.TSNE(perplexity=40, n_components=2, init='pca')X = pca.fit_transform(X)## create dtfdtf = pd.DataFrame()for k,v in dic_clusters.items():    size = len(dtf) + len(v)    dtf_group = pd.DataFrame(X[len(dtf):size], columns=["x","y"],                              index=v)    dtf_group["cluster"] = k    dtf = dtf.append(dtf_group)        ## plotfig, ax = plt.subplots()sns.scatterplot(data=dtf, x="x", y="y", hue="cluster", ax=ax)ax.legend().texts[0].set_text(None)ax.set(xlabel=None, ylabel=None, xticks=[], xticklabels=[],        yticks=[], yticklabels=[])for i in range(len(dtf)):    ax.annotate(dtf.index[i],                xy=(dtf["x"].iloc[i],dtf["y"].iloc[i]),                xytext=(5,2), textcoords='offset points',                ha='right', va='bottom')
Cool, they look isolated enough from each other. The Entertainment cluster is closer to the Tech one than the Politics one, which makes sense as words like “apple” and “youtube” can appear in both Tech and Entertainment news.
It’s time to embed the corpus we preprocessed and the target clusters we created in the same vector space. Basically, we’re doing this:
Yes, I’m using BERT for this. It’s true that you could utilize any Word Embedding model (i.e. Word2Vec, Glove, ...), even the one that we already loaded to define keywords, so why bother to use such a heavy and complex language model? That’s because BERT doesn’t apply a fixed embedding, instead it looks at the entire sentence and then assigns an embedding to each word. Therefore, the vector BERT assigns to a word is a function of the entire sentence, so that a word can have different vectors based on the contexts.
I’m going to load the original pre-trained version of BERT with the package transformers and give an example of the dynamic embedding:
tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-            uncased', do_lower_case=True)nlp = transformers.TFBertModel.from_pretrained('bert-base-uncased')
Let’s use the model to convert the string “river bank” into vectors and print the one assigned to the word “bank”:
txt = "river bank"## tokenizeidx = tokenizer.encode(txt)print("tokens:", tokenizer.convert_ids_to_tokens(idx))print("ids   :", tokenizer.encode(txt))## word embeddingidx = np.array(idx)[None,:]embedding = nlp(idx)print("shape:", embedding[0][0].shape)## vector of the second input wordembedding[0][0][2]
If you do the same for the string “financial bank”, you’ll see that the vector assigned to the word “bank” is different because of the context. Please note that the BERT tokenizer inserts special tokens at the beginning and end of sentences and its vector space has a dimension of 768 (to understand better how BERT processes text you can check this article).
Having said that, the plan is to use BERT Word Embedding to represent each text with an array (shape: number of tokens x 768) and then summarize each article into a mean vector.
So the final feature matrix will be an array with shape: number of documents (or mean vectors) x 768.
## function to applydef utils_bert_embedding(txt, tokenizer, nlp):    idx = tokenizer.encode(txt)    idx = np.array(idx)[None,:]      embedding = nlp(idx)    X = np.array(embedding[0][0][1:-1])    return X## create list of news vectorlst_mean_vecs = [utils_bert_embedding(txt, tokenizer, nlp).mean(0)                  for txt in dtf["text_clean"]]## create the feature matrix (n news x 768)X = np.array(lst_mean_vecs)
We can do the same with the keywords in the target clusters. In fact, each label is identified by a list of words that help BERT to understand the context within the clusters. Hence, I’m going to create a dictionary label : cluster mean vector.
dic_y = {k:utils_bert_embedding(v, tokenizer, nlp).mean(0) for k,v         in dic_clusters.items()}
We started with just some text data and 3 strings (“Entertainment”, “Politics”, “Tech”) and now we have a feature matrix and a target variable... ish.
Finally, it’s time to build a model that classifies the news based on the similarity to each target cluster.
I am going to use Cosine Similarity, a measure of similarity based on the cosine of the angle between two non-zero vectors, which equals the inner product of the same vectors normalized to both have length 1. You can easily use the cosine similarity implementation of scikit-learn, which takes 2 arrays (or vectors) and returns an array of scores (or a single score). In this case, the output is going to be a matrix with shape: number of news x number of labels (3, Entertainment/Politics/Tech). To put it another way, each row will represent an article and contain one similarity score for each target cluster.
In order to run the usual evaluation metrics (Accuracy, AUC, Precision, Recall, ...), we have to rescale the scores in each row so that they sum to 1 and decide a category to label the article with. I’m going to choose the one with the highest score, but it could be wise to set some minimum thresholds and leave out predictions with really low scores.
#--- Model Algorithm ---### compute cosine similaritiessimilarities = np.array(            [metrics.pairwise.cosine_similarity(X, y).T.tolist()[0]              for y in dic_y.values()]            ).T## adjust and rescalelabels = list(dic_y.keys())for i in range(len(similarities)):    ### assign randomly if there is no similarity    if sum(similarities[i]) == 0:       similarities[i] = [0]*len(labels)       similarities[i][np.random.choice(range(len(labels)))] = 1    ### rescale so they sum = 1    similarities[i] = similarities[i] / sum(similarities[i])## classify the label with highest similarity scorepredicted_prob = similaritiespredicted = [labels[np.argmax(pred)] for pred in predicted_prob]
Just like in classic supervised use cases, we have an object with predicted probabilities (here they’re adjusted similarity scores) and another with predicted labels. Let’s check how we did:
y_test = dtf["y"].valuesclasses = np.unique(y_test)y_test_array = pd.get_dummies(y_test, drop_first=False).values## Accuracy, Precision, Recallaccuracy = metrics.accuracy_score(y_test, predicted)auc = metrics.roc_auc_score(y_test, predicted_prob,                             multi_class="ovr")print("Accuracy:",  round(accuracy,2))print("Auc:", round(auc,2))print("Detail:")print(metrics.classification_report(y_test, predicted))    ## Plot confusion matrixcm = metrics.confusion_matrix(y_test, predicted)fig, ax = plt.subplots()sns.heatmap(cm, annot=True, fmt='d', ax=ax, cmap=plt.cm.Blues,             cbar=False)ax.set(xlabel="Pred", ylabel="True", xticklabels=classes,        yticklabels=classes, title="Confusion matrix")plt.yticks(rotation=0)fig, ax = plt.subplots(nrows=1, ncols=2)## Plot rocfor i in range(len(classes)):    fpr, tpr, thresholds = metrics.roc_curve(y_test_array[:,i],                             predicted_prob[:,i])    ax[0].plot(fpr, tpr, lw=3,               label='{0} (area={1:0.2f})'.format(classes[i],                               metrics.auc(fpr, tpr))               )ax[0].plot([0,1], [0,1], color='navy', lw=3, linestyle='--')ax[0].set(xlim=[-0.05,1.0], ylim=[0.0,1.05],           xlabel='False Positive Rate',           ylabel="True Positive Rate (Recall)",           title="Receiver operating characteristic")ax[0].legend(loc="lower right")ax[0].grid(True)    ## Plot precision-recall curvefor i in range(len(classes)):    precision, recall, thresholds = metrics.precision_recall_curve(                 y_test_array[:,i], predicted_prob[:,i])    ax[1].plot(recall, precision, lw=3,                label='{0} (area={1:0.2f})'.format(classes[i],                                   metrics.auc(recall, precision))              )ax[1].set(xlim=[0.0,1.05], ylim=[0.0,1.05], xlabel='Recall',           ylabel="Precision", title="Precision-Recall curve")ax[1].legend(loc="best")ax[1].grid(True)plt.show()
Okay, I’m the first to say that it’s not the best Accuracy I’ve ever seen. On the other hand, it’s not bad at all considering that we didn’t train any model and we even made up the target variable. The main issue is over 4k Politics observations classified as Entertainment, but these performances can be easily improved by fine-tuning the keywords for those two categories.
Let’s try to understand what led our algorithm to classify news with a category instead of the others. Let’s take a random observation from the corpus:
i = 7txt_instance = dtf["text_clean"].iloc[i]print("True:", y_test[i], "--> Pred:", predicted[i], "|       Similarity:", round(np.max(predicted_prob[i]),2))print(txt_instance)
It’s a Politics observation properly classified. Probably, the words “republican” and “clinton” gave BERT the right hint. I will visualize the mean vector of the article in a 2D space and plot the top similarities with the target cluster.
## create embedding Matrixy = np.concatenate([embedding_bert(v, tokenizer, nlp) for v in                     dic_clusters.values()])X = embedding_bert(txt_instance, tokenizer,                   nlp).mean(0).reshape(1,-1)M = np.concatenate([y,X])## pcapca = manifold.TSNE(perplexity=40, n_components=2, init='pca')M = pca.fit_transform(M)y, X = M[:len(y)], M[len(y):]## create dtf clustersdtf = pd.DataFrame()for k,v in dic_clusters.items():    size = len(dtf) + len(v)    dtf_group = pd.DataFrame(y[len(dtf):size], columns=["x","y"],                              index=v)    dtf_group["cluster"] = k    dtf = dtf.append(dtf_group)## plot clustersfig, ax = plt.subplots()sns.scatterplot(data=dtf, x="x", y="y", hue="cluster", ax=ax)ax.legend().texts[0].set_text(None)ax.set(xlabel=None, ylabel=None, xticks=[], xticklabels=[],        yticks=[], yticklabels=[])for i in range(len(dtf)):    ax.annotate(dtf.index[i],                xy=(dtf["x"].iloc[i],dtf["y"].iloc[i]),                xytext=(5,2), textcoords='offset points',                ha='right', va='bottom')## add txt_instanceax.scatter(x=X[0][0], y=X[0][1], c="red", linewidth=10)           ax.annotate("x", xy=(X[0][0],X[0][1]),            ha='center', va='center', fontsize=25)## calculate similaritysim_matrix = metrics.pairwise.cosine_similarity(X, y)## add top similarityfor row in range(sim_matrix.shape[0]):    ### sorted {keyword:score}    dic_sim = {n:sim_matrix[row][n] for n in                range(sim_matrix.shape[1])}    dic_sim = {k:v for k,v in sorted(dic_sim.items(),                 key=lambda item:item[1], reverse=True)}    ### plot lines    for k in dict(list(dic_sim.items())[0:5]).keys():        p1 = [X[row][0], X[row][1]]        p2 = [y[k][0], y[k][1]]        ax.plot([p1[0],p2[0]], [p1[1],p2[1]], c="red", alpha=0.5)plt.show()
Let’s zoom a bit on the cluster of interest:
Overall, we can say the mean vector is pretty similar to the Politics cluster. Let’s break down the article into tokens to see which ones “activated” the right cluster.
## create embedding Matrixy = np.concatenate([embedding_bert(v, tokenizer, nlp) for v in                     dic_clusters.values()])X = embedding_bert(txt_instance, tokenizer,                   nlp).mean(0).reshape(1,-1)M = np.concatenate([y,X])## pcapca = manifold.TSNE(perplexity=40, n_components=2, init='pca')M = pca.fit_transform(M)y, X = M[:len(y)], M[len(y):]## create dtf clustersdtf = pd.DataFrame()for k,v in dic_clusters.items():    size = len(dtf) + len(v)    dtf_group = pd.DataFrame(y[len(dtf):size], columns=["x","y"],                              index=v)    dtf_group["cluster"] = k    dtf = dtf.append(dtf_group)## add txt_instancetokens = tokenizer.convert_ids_to_tokens(               tokenizer.encode(txt_instance))[1:-1]dtf = pd.DataFrame(X, columns=["x","y"], index=tokens)dtf = dtf[~dtf.index.str.contains("#")]dtf = dtf[dtf.index.str.len() > 1]X = dtf.valuesax.scatter(x=dtf["x"], y=dtf["y"], c="red")for i in range(len(dtf)):     ax.annotate(dtf.index[i],                  xy=(dtf["x"].iloc[i],dtf["y"].iloc[i]),                  xytext=(5,2), textcoords='offset points',                  ha='right', va='bottom')## calculate similaritysim_matrix = metrics.pairwise.cosine_similarity(X, y)## add top similarityfor row in range(sim_matrix.shape[0]):    ### sorted {keyword:score}    dic_sim = {n:sim_matrix[row][n] for n in                range(sim_matrix.shape[1])}    dic_sim = {k:v for k,v in sorted(dic_sim.items(),                 key=lambda item:item[1], reverse=True)}    ### plot lines    for k in dict(list(dic_sim.items())[0:5]).keys():        p1 = [X[row][0], X[row][1]]        p2 = [y[k][0], y[k][1]]        ax.plot([p1[0],p2[0]], [p1[1],p2[1]], c="red", alpha=0.5)plt.show()
As we thought, there are words in the text which are clearly linked to the Politics cluster, but some others are more similar to the Entertainment general context.
This article has been a tutorial to demonstrate how to perform text classification when a labeled training set isn't available.
I used a pre-trained Word Embedding model to build a set of keywords to contextualize the target variable. Then I transformed those words and the corpus in the same vector space with the pre-trained BERT language model. Finally, I calculated the Cosine Similarity between text and keywords to determine the context of each article and I used that information to label the news.
This strategy isn’t the most effective but it’s definitely efficient as it allows you to deliver good results quickly. Moreover, this algorithm can be used as a baseline for a supervised model, once a labeled dataset is obtained.
I hope you enjoyed it! Feel free to contact me for questions and feedback or just to share your interesting projects.
👉 Let’s Connect 👈
This article is part of the series NLP with Python, see also: