Explore Spiritual World Using NLP (Natural Language Processing)
By: Priyanka Dave
Tags: NLP, Sentiment Analysis, Topic Modeling, Information Retrieval, Text Summarization

7. Fetch Similar Documents Using Search Query:

What is document similarity?

When we want to compute similarity based on meaning, we call it semantic text similarity . Due to the complexities of natural language, this is a very complex task to accomplish.

. Due to the complexities of natural language, this is a very complex task to accomplish. Another methods works on lexical level , that is, using only the words in the sentence. This is the traditional approach to compute text similarity between documents.

, that is, using only the words in the sentence. This is the traditional approach to compute text similarity between documents. Here we will explore traditional method to find similar documents.

How to find similar documents?

Select a way of transforming documents into vectors. Select similarity measure for vectors. Compute distance between search query and all the documents using selected similarity measure and fetch the documents with smallest distance.

Common similarity measures:

Jaccard distance Cosine distance Euclidean distance

Here we will use Euclidean distance to find similar documents.

What is Euclidean Distance?

Euclidean distance is the distance between 2 points in a multidimensional space. Closer points are more similar to each other. Further points are more different from each other.

Search Query to fetch similar documents from both the datasets:

Search Query

Output from Articles dataset:

Topic KeyWords: ['30th birthday', 'authentic self', 'create space', 'expand feel', 'expect experience', 'experience coming', 'experience deep', 'experiment play', 'explore depth', 'feel allow'] Topic Prob Scores of text: [[0.6 0.1 0.1 0.1 0.1 0.1 0.1]] Most Similar Doc's Probs: [[1. 0. 0. 0. 0. 0. 0.]] Document:

"When Saturn is involved, think: responsibilities, rules, restrictions, obligations, challenges, and lessons. Sometimes tough ones. It’s like a cosmic check-in—and check-up—from the Lord of Karma himself. This is your astrological coming-of-age story. It is important to remember that Saturn’s job is to show you the areas of your life where you are the hardest on yourself. You are being pushed to level up to the next stage of life. After all, Saturn is the planetary taskmaster, and at this time, he will likely hold your feet to the proverbial fire. Consider your life before this moment: You have been restricted by the rules placed on you by your parents, teachers, bosses, mentors, and even society at large. It is a slow-moving process, so as you go through this, remember that there are no shortcuts, there will be tests, and you will get through it. By the end of your Saturn return (and by the time of your second and third Saturn return), as you continue to apply these hard-earned lessons, you will eventually greet Saturn like an old friend. You’ll see that Saturn’s tough love during your return called upon you to stop giving away your power. As a result, you’ll see that Saturn was not the author of your life. You are—and always were. Subsequently, Saturn didn’t take away anything that was meant for you. Saturn is here to confront you and ensure that you’re on the right path."

Output from YouTube Comments dataset:

Topic KeyWords: ['work hard', 'hard work', 'lord shiva', 'past karma', 'self realisation', 'negative thought', 'decide want', 'person self', 'realise mistake', 'deed past'] Topic Prob Scores of text: [[0.1 0.1 0.1 0.1 0.6 0.1 0.1]] Most Similar Doc's Probs: [[0.1 0.1 0.1 0.1 0.6 0.1 0.1]] Document:

"you can do everything man!! but now u really doing your best.... of course its all depend on karmas but finally you listened your soul which must be key ruling in your chart which is sun !! your fame is near to success for sure. tnx"

Code To Fetch Similar Documents

8. Text Summarization:

What is Text Summarization?

Text summarization is the process of creating a short and accurate summary of a longer text document.

Here we will explore Bert Extractive Summarizer.

Why BERT?

Because BERT is the best Text Summarizer till date.

What is BERT?

Bidirectional Encoder Representations from Transformers is pretrained language model.

is pretrained language model. BERT Summarizer works by first embedding the sentences, then running a clustering algorithm, finding the sentences that are closest to the cluster’s centroids.

You will be able to install BERT text summarizer pretrained model using,

pip install bert-extractive-summarizer

And then by writing below 3 lines of code, you will be able to generate summary of your long text documents.

from summarizer import Summarizer

model = Summarizer()

summary = model(text)

You can play with different parameters of model and check output.

ratio: Ratio of sentences to summarize to from the original body. min_length: The minimum length to accept as a sentence. max_length: The maximum length to accept as a sentence. num_sentences: Number sentences require in generated summary. calculate_optimal_k: To find optimum number of sentences.

You will be able to explore more parameters here.

Let’s generate summary of our datasets.

Generate summary by finding optimum number of sentences using calculate_optimal_k:

youtube_comments_summary:

Honestly, at 28-31 if you dont feel the need to make a shift in your life there's something wrong. I'm already extremely Saturnian... I get no enjoyment out of life, I just go through the motions and survive. Please tell me how the PLANETS effect the SIDERAL SIGN OPHIUCHUS Wow so if I understand this correctly I have a well placed Saturn. Almost all of what you said has actually come true with me... I am unfortunate enough to be born with my sun moon and rising planet all in Scorpio... And right now i'm going through hell... Wondering if you could tell me what more backlashes to expect Sir , Can you please explain , If shani Dev in tungi , n its sade sati effect With house-placement starting 1st house the sign of the Asc.

articles_summary:

During your Saturn return, you are a teenager and the planet of discipline is your dad, barging into your room checking to see if you’ve done your homework. Some old souls that made wise choices from the get-go may find it to be a passage where things are solidified. Maybe you're comfortable, but not exactly happy, in your relationship. Because Saturn is one of the teacher planets in the sky, when it resets in our birth chart, its a cosmic indication that it's time to learn a lesson.

2. Generate summary using ratio:

youtube_comments_summary_ratio:

Honestly, at 28-31 if you dont feel the need to make a shift in your life there's something wrong. Mines occur in 2022...and I'm 26 getting my life together now..going to school and I feel I have to I have no choice so I can do it..my birth chart says my Saturn is in the 8th house but I was told its in the 9th idk Can someone explain what does he mean by damaged Moon or Mercury? Strong aspects of Jupiter to all planets... I'm doing my best at keeping soft and strong. Ketu with saturn confuses me, its like, am I supposed to "let go" of saturnian things?.............

articles_summary_ratio:

During your Saturn return, you are a teenager and the planet of discipline is your dad, barging into your room checking to see if you’ve done your homework. The Saturn Return is a wake-up call, and this is why so many fear it's sobering realities. Some "old souls" that made wise choices from the get-go may find it to be a passage where things are solidified. Saturn will give you very clear signals if you're not aligned with your calling and pursuing your passions. When you demonstrate your commitment to your goal by taking one concrete step in that direction, Saturn will reward your effort. No matter what your family has to say about it.) Saturn returns and oppositions can be extremely powerful and turn your life in a completely new direction. Because Saturn is one of the "teacher planets in the sky," when it resets in our birth chart, it's a cosmic indication that it's time to learn a lesson.

9. Conclusion:

Here we have discussed 4 NLP techniques:

Sentiment Analysis Topic Modeling Document Similarity Text Summarization

Output from all the techniques was much similar for both the datasets.

In sentiment analysis we have seen that the ratio of positive comments/articles was lesser as compared to other 2 categories. — We can related this with the fact that “Saturn is considered as malefic planet.”

we have seen that the ratio of positive comments/articles was lesser as compared to other 2 categories. — We can related this with the fact that In articles, Subjectivity & Objectivity was divided equally as articles/blogs are generally written based on one’s knowledge or opinion. But Objectivity was higher in YouTube Comments as most of them are written based on one’s real life experiences.

& was divided equally as articles/blogs are generally written based on one’s knowledge or opinion. But was higher in YouTube Comments as most of them are written based on one’s real life experiences. Output of Topic Modeling shows that the most discussed topics of both the datasets are “hard work”, “learn lesson”, “karma”. — Saturn’s lessons are discipline, hard work, and sacrifice. Saturn is also known as the Lord or Karma and Justice.

So, here we can conclude that we have successfully gained some meaningful insights from both the datasets using NLP techniques.