Sentence Scoring Based on Noun and Numerical Values | by Mofiz Mojib Haider | Towards Data Science

Sentence scoring is one of the most used processes in the area of Natural Language Processing (NLP) while working on textual data. It is a process to associate a numerical value with a sentence based on the used algorithm’s priority.
This process is highly used especially on text summarization. There are many popular methods for sentence scoring like TF-IDF, TextRank and so on. Here, we are going to examine a new approach to score a sentence based on the occurrence of nouns, numerical values, and word2vec mean similarity.
Step-1: Import libraries
Step-2: Text processing and methods
A random paragraph from a random news article from BBC has been selected to examine. Processing steps are mentioned below:
Split each of the sentences using sentence tokenizer from NLTK sent_tokenize.
Split each of the sentences using sentence tokenizer from NLTK sent_tokenize.
2. Special characters like %, $, #, @, etc has been removed.
3. All the words of each sentence have been tokenized.
4. Stop words like and, but, or, which are included in the stop word list has been removed.
5. It has been ensured that all the words of a sentence has only one occurrence.
6. Lemmatization has been used to find the root word
Step-3: Gensim Word2Vec
Make a vector list of the lemmatized text list using the Gensim Word2Vec skip-gram method.
Step-4: Sentence Scoring
To score a sentence two methods have been used named meanOfWord() and checkNum(). This method has two lists of pos values which include nouns and digits. But the problem is numbers like ‘ 234’, ‘ 34.5’ (blank space before the number) considered as a noun in NLTK pos_tag(). So, another method has been used checkNum() which confirms its a number or not. The procedure used in meanOfWord() has mentioned below:
Find the similarity of each word from the word2vec model and calculate the mean similarityCheck the word if it is a number or notIf number then add 1 with the mean similarity if noun then adds 0.25 with the mean similarityReturn the updated mean value
Find the similarity of each word from the word2vec model and calculate the mean similarity
Check the word if it is a number or not
If number then add 1 with the mean similarity if noun then adds 0.25 with the mean similarity
Return the updated mean value
For each sentence from the lemmatized list, the mean score has been calculated and pushed into the score list. The output of the score will be the index number of the sentence and the numerical score of that sentence.
My output is:
[[0, 2.875715106488629], [1, 3.3718763930364872], [2,2.117822954338044], [3, 4.115576311542342]]
which indicates:
'China remains the main area of concern for BMW after sales there fell 16% last year.' got 2.875715106488629 score.'However, BMW is hopeful of a much better year in 2005 as its direct investment in China begins to pay dividends.' got 3.3718763930364872 score.'The company only began assembling luxury high-powered sedans in China in 2003.' got 2.117822954338044 score.'2004 was generally a good year for BMW, which saw revenues from its core car-making operations rise 11%.' got 4.115576311542342 score.
The 4th sentence has got the highest score because it contains more nouns and numerical values than other sentences. It may vary, as the word2vec model may generate a different vector for different iteration.
The complete code is given as follows:
Caution:
This approach will give a better performance where the amount of nouns and numerical values is very much higher.
If you find this approach useful then you can use it and share it with others. Feel free to write suggestions on the comment section.