Key Phrase Extraction & Applause Prediction
By: Krishna Yadav
Tags: Data Mining, Naturallanguageprocessing, Text Classification, Deep Learning, Machine Learning

*Auto ML tool for performing ML techniques on raw data files of medium online article

Best Project Award Winner in Machine Learning @IIITD 2019–2020

Abstract — With the increase in the availability of content over the internet, it is very difficult to get noticed. It has become an utmost priority of the blog writers to get some feedback over their creations to be confident about the impact of their article. We are training a machine learning model to learn popular article styles, in the form of vector space representations using various word embeddings, and their popularity based on claps and tags.

Motivation

Today’s world is full of online content and a lot more is produced each day. Medium is one of the top platforms for spreading information and knowledge about almost any domain in society. It is abundantly used to publish content, articles on computer science domains such as ML, AI, Data Engineering, etc. Getting popularity over internet.

2. Introduction

Article writing and gaining popularity is a difficult task. The problem comes for the new authors who don’t have much idea about how to get popularity. We have developed intuition into making an interesting and interesting Medium article. Instead of purely relying on a guess theory, we wanted to test our theory using data science and machine learning. As a Data Scientist, we have always been curious about creativity and automation. Also, we believe that our article and hard work on this model combines both faces of science to predict the best Medium article. We introduce our work and further work Platforms like Medium have their own metrics to evaluate the content by allowing readers to clap to show their appreciation. Also, these platforms offer provision to add a few Key-Phrases that best provide the subject of the content.

Now, it becomes a necessity for the writer to get some assistance for the best possible key-phrases to get the articles ranked up and tentative popularity of the new article. Our aim in this project will be to train a model on the Medium.

3. Literature Survey

Key phrase extraction is a big field and a lot of people have worked on it already. We have many pre-trained models like Bert and pke rake etc. that help to extract the key phrases. Although we have many advancements in text classification, this whole problem of article performance problem is explored by only a smaller group of researchers. pke is an open-source python-based keyphrase extraction toolkit. It provides an end-to-end keyphrase extraction pipeline in which each component can be easily modified or extended to develop new models. pke also allows for easy benchmarking of state-of-the-art keyphrase extraction models, and ships with supervised models trained on the SemEval-2010 dataset. [1], [2]. Sahrawat et al. [8] have used neural network models such as bert and bi-lstm to extract key phrases from scholarly articles as a sequence labeling task solved using a BiLSTM-CRF, where the words in the input text are represented using deep contextualized embeddings.

4. Dataset Used

The dataset that we used for this project is being scrapped using selenium in the form of JSON strings from the official Medium website. So far, we have over 2574 articles of diverse popularity. We are collecting more data but since the annotation verification takes time so will conclude this in the final presentation. For pre-processing, we removed all the Non_ASCII characters and the white-spaces or special characters. then we converted the digits into their English equivalent. followed by spell correction and lemmatization.

Our data had a lot of new articles having zero claps so we removed those to avoid misdirection. and finally took doc2vec embeddings to get a numerical document vector.

5. Data Scraping

Web Scraping also called “Crawling” or “Spidering” is the technique to gather data automatically from an online source usually from a website. While Web Scraping is an easy way to get a large volume of data in a relatively short time frame, it adds stress to the server where the source is hosted. We scrapped the data from the medium using selenium. Data mining or gathering data is a very primitive step in the data science life cycle. As per business requirements, one may have to gather data from sources like SAP servers, logs, Databases, APIs, online repositories, or web. Tools for web scraping like Selenium can scrape a large volume of data such as text and images in a relatively short time.

Selenium is an open-source web-based automation tool. Selenium is primarily used for testing in the industry but It can also be used for web scraping. We’ll use the Chrome browser but you can try it on any browser, It’s almost the same. [5] We scrapped the by following steps:

Search URLs of articles with the given set of tags to capture domain related articles only.

Scrapping the useful data from each webpage.

Saving the scraped article data in json object.

6. Pre-processing

The steps for natural language preprocessing are taken up to convert each sentence into equivalent processed word representation [1], [2], [7], [8].

The techniques used to process each word are:

Punctuation Removal: ‘.”,?~!<@#$%&*) Whitespaces Removal:

\t \a Non-Ascii Characters: non-keyboard, emoji Tokenization: ‘hello’, ‘world’ Number conversion: 1 to ‘one’ Stop-word removal: the, are, is Spelling correction: thro, teling, forgt Word segmentation: haveto, whynot Lemmatization: mice to mouse Expanding abbreviations: I’ve, can’t Scaling: [1, 90,99] to [0.1, 0.9, 0.99]

Text data is not suitable to feed into ML models, so we used Glove and doc2vec combined with sequences obtained by the text_to_sequence method from Keras’ tokenizer.

8. Feature Extraction

After generalizing the length of the questions, we are left with long vectors with a lot of relevant information (be it more or less). Since the length is directly proportional to the vocab size, as mentioned in Zip’s law [10], [11], it is crucial to extract interesting features only. We tried PCA and LDA feature extraction[12]. Word embeddings also helped and finally got 23 top features to apply ML algorithms. More or fewer features than 23 lead us to the reduced performance of our model. That’s why we are only capturing the most relevant information with this set of features only. Dimensionality reduction also helps to visualize the top data components for better understanding, as shown in Figure 3. For extracting the features out of textual data of blog articles we have used doc2vec which will be able to generate a vector for corresponding documents(a complete article). We used RFE with chi2 to finally rank features on the basis of their importance.

9. Proposed Model

We tried several approaches and combinations to reach the final set of conclusions. However, the pre-processing steps for all remain the same. Pre-processing differs for regression tasks and keyword prediction tasks.

9.1 Applause Prediction

We have used Linear Regression for regression over the claps and we also proposed the use of KNN to find the closest similar data points to our test document in vector space. Regression gives MSE i.e. we calculated the deviation of the predicted results from actual ground truth results. This is to get the factor of claps at each document. The initial MSE we got was around 13000 which was huge while using a Doc2vec as the features. Using KNN for regression we will be able to reduce MSE further to approx 4000 and an accuracy of 37%.For this, we have around 2345 medium articles.

We slightly modified our task from a regression on claps to popularity score predictions as we are unable to reduce the mse further.

We tried to scrap more data. So the final dataset size is now 7421 articles. We updated the label values on the range of 1–5 in which 1 being the least popular and 5 being the most. The score assignment has been done with the help of the applause counts. So its become the multilabel classification task with five classes in which we are able to get the accuracy of 48%.For a binary classification task of popular non-popular and popular where we have assigned labels not popular for a number of claps under 50 and popular otherwise we were able to get the accuracy of 94%.

In Applause Prediction using linear regression and knn we were able to reduce our loss around 4000 mse.

Which is still huge! Right? So we slightly modify the task to get the popularity score prediction of a given article. We divide the applauses into five buckets of the score(1–5).

Then we further try to experiment by assigning just two labels 0 and 1 for popular or non-popular respectively.

9.2 Key Phrase Extraction

BIO-Tagging Scheme

9.2.1 Deep Learning Models

Deep Learning has been proven to solve many hard problems. We have planned to use deeply learned models like RoBERTa to get optimized results for our model final prediction for the article being popular or not. We are planning to use BIO-Tagging to represent each document for keyphrase extraction. It is a common format the similar kinds of NLP tasks such as NER. And later using deep neural networks to predict those BIO tags as labels.

Deep Learning Models (Best)

10. Performance Improvement

What Improved our Performance, and why?

TfIdf: performs better than doc2vec as it captures sublinear relationships. Stop-words: In our case, stop words are worthless as simply misdirect the classification as they occur very often in every document. Dimension Reduction works: our vector representation has a lot of features and feeding that to a model increases time and hides essential information. Adding features: Selected word embeddings give us 23 sufficient features. Adding or removing features reduces the performance as the relevant information gets compromised. scaling the feature helps to get better accuracy especially in case of svm. Smote: Smote helps resolving the data imbalancing.Which in turn helps in performance. Converting the task : Converting the task of applause prediction into 1–5 score classification. As Classification models are able to get better decision boundaries than regression models’ best fit line.

Model Architecture and Background Story

What didn’t work, and why?

Spelling Correction: Most of the questions didn’t have any spelling mistakes. It is assumed that the user is aware of the language. Scaling: Vector representation of questions is already scaled. Further scaling doesn’t help anymore. All embeddings: Embeddings usually capture most of the information. Combining gensim embeddings doesn’t solve the purpose. Adding features: Selected word embeddings give us 23 sufficient features. Adding or removing features reduces the performance as the relevant information gets compromised. Reducing Vector size: It improves the training time but at the cost reduced performance. Increasing epochs: Increasing epochs for doc2vec training fails to give any better features. Varying Regression: Changing naive regression to lasso,ridge aur elastic fails to give better performance. Hyperparameter Tuning: As data itself is biased, so hyperparameter failed to improve performance to a significant extent.

11. Results

For the regression task, we used linear regression to draw a linear decision boundary and estimate the claps of new articles. Also, we propose the use of knn in terms of document similarity to find k nearest documents to the new document in multidimensional vector space and take an average of those document claps. Tuning the hyperparameters showed a lot of improvement. We are planning task-specific fine-tuning for both of these tasks.

11.1 Applause Prediction

The results are obtained after passing the model through rigorous task-specific fine-tuning. Kernel, activation, depth, smoothening parameter, number of iterations, overfitting and underfitting regularizers are a few to name.

Applause Prediction Performace

After including features such as authors, reading time, number of images in the article we able to reduce mse.

In the multi-class task, we will able to get an accuracy of 0.48.

In the popularity task, the final best accuracy we got is 0.94.

Loss and accuracy depend Linearly on the amount of data.

Having multiple approaches made our app reach a wider audience since not everyone wants to get into granular details as results.

11.2 KeyPhrase Extraction

For the Key-Phrase Extraction task, we initialized the Topic-Rank keyphrase extraction model and carried the preprocessing using spacy. The key phrase candidate selection happens as sequences of nouns and adjectives (i.e. `(Noun|Adj)*`). Then candidate weighting happens using a random walk algorithm. And finally, N-best key phrases i.e. highest scored candidates are extracted as (keyphrase, score) tuples.

DeepLearning has its own benefits. It may overfit sometimes but rigorous fine-tuning gave us promising results. For keyphrase extraction, Bidirectional LSTM trained over tfidf embeddings produced promising results. This task is highly unbalanced so we reported f1-score based metrics.

12. User Interface

We made an interactive web application based on HTML, python, flask, and JQuery using ajax API call to retrieve top k documents from the collection of best performing models. It has two modes of operation i.e. text and audio(7).

User Interface for Audio and Textual I/O

Text: Input from keyboard

Code Snippet for Audio Interface

Audio: voice interaction with the web app via default microphone and speakers. Users can see the top 5 results in the web app screen and allow the app to store k results in the ’temp.txt’ file for later reference. Figure 6 and 7 shows the user interface of the created web application for the query ’Our performance in cricket’ and ‘Ronaldo’ respectively.

TTS is a great tool to improve the impact of your user experience this can be incorporated in your interface with these steps.

15. Conclusion

We started with the aim to offer an end to end solution for budding researchers and let them write successful articles. This system would take in input as text and gives out whether the author is up to the mark as per the current trend. Deep-Learning has it’s own benefits. It may overfit sometimes but rigorous fine-tuning gave us promising results. For keyphrase extraction, Bidirectional LSTM trained over tfidf embeddings produced promising results. This task is highly unbalanced so we reported f1-score based metrics.

14. Future Work

The key problem in this task is that data is not linearly separable and it has a lot of overlapping. So instead of taking specific domains of medium blogs if we go for the general articles, it would be more beneficial. Another problem is that the dataset is highly skewed towards the claps 0,1 as articles with good clap count are very few. We initially proposed KNN to predict the number of claps but still after hyperparameter tuning and feature engineering we failed to get any better mse. In the regression of applause, we plan to incorporate other features with more weightage i.e. date of publishing, author name, etc. We are going to consider contextualized word embedding using deep learning models to capture the context for embedding.

15. Authors’ Contributions

Krishna Yadav (MT19039) [LinkedIn]

Data cleaning and Preprocessing Data Annotation Data Analysis and visualization Key Phrase Extraction Baseline Results Deep Learning Models Analysis and conclusion

Lakshya Choudhary (MT19067) [LinkedIn]

Data Crawler Data Annotation Feature selection LR, KNN | Parameter Tuning Baseline Results Fine-tuned ML models Analysis and conclusion

References

[1] https://github.com/boudinfl/pke.

[2] https://arxiv.org/abs/1810.04805.

[3] Z. Liu, X. Lv, K. Liu, and S. Shi, “Study on SVM compared with the other text classification methods,” in 2nd International Workshop on Education Technology and Computer Science, ETCS 2010, 2010, vol. 1, pp. 219–222, doi: 10.1109/ETCS.2010.248.

[4] https://monkeylearn.com/keyword-extraction/.

[5] https://www.analyticsvidhya.com/blog/2020/08/web-scraping-selenium-with-python/.

[6] M. S. Phyu and K. T. Nwet, “A Study on a Joint Deep Learning Model for Myanmar Text Classification,” in 2020 IEEE Conference on Computer Applications, ICCA 2020, Feb. 2020, doi: 10.1109/ICCA49400.2020.9022809.

[7] T. Bi, P. Liang, A. Tang, and C. Yang, “A systematic mapping study on text analysis techniques in software architecture,” J. Syst. Softw., vol. 144, no. August, pp. 533–558, 2018, doi: 10.1016/j.jss.2018.07.055.

[8] Sahrawat D. et al. (2020) Keyphrase Extraction as Sequence Labeling Using Contextualized Embeddings. In: Jose J. et al. (eds) Advances in Information Retrieval. ECIR 2020. Lecture Notes in Computer Science, vol 12036. Springer, Cham. https://doi.org/10.1007/978-3-030-45442-5_41.

[9] https://towardsdatascience.com/extracting-keyphrases-from-text-rake-and-gensim-in-python-eefd0fad582f.

[10] S. T. Piantadosi, “Zipf’s word frequency law in natural language: A critical review and future directions,” Psychon. Bull. Rev., vol. 21, no. 5, pp. 1112–1130, Oct. 2014, doi: 10.3758/s13423–014–0585–6.

[11] L. Adamic, B. H.- Glottometrics, and undefined 2002, “Zipf’s law and the Internet.” Accessed: May 13, 2020. [Online]. Available: http://www.ram-verlag.de.

[12] X. Yan, J. Guo, Y. Lan, and X. Cheng, “A biterm topic model for short texts,” WWW 2013 — Proc. 22nd Int. Conf. World Wide Web, pp. 1445–1455, 2013, doi: 10.1145/2488388.2488514.

Acknowledgments