NLP Profiler: Profiling datasets with one or more text columns | by Himanshu Sharma | Towards Data Science

Have you ever worked with a large text dataset, if you have then you must know how difficult it is to analyze text data, you need to use different functionalities of Natural Language Processing in order to get an idea of the data you are working with? But if I tell you that now don’t need to worry about performing a number of operations in order to analyze text data then it would be great right.
So let’s get started with NLP Profiler, which is a python tool/package/module used to create a summary of the data similar to that created by using describe() function of pandas for numerical data in a dataframe. It allows you to create a profile of textual datasets and analyze different properties of the text data. It generates high-level insights into the data you are using.
In this article, I will take you through different functionalities of NLP Profiler and I will show you how we can use that. We will be working on this using Google Collab.
We will start by installing NLP Profiler by the command given below. As we are using Google Collab so you need to copy this command and run it in your collab notebook to install the NLP profiler.
!pip install nlp_profiler
In order to start exploring NLP Profiler let us start by connecting our collab notebook with Kaggle, this is important because we will directly load the dataset from Kaggle to the collab notebook. Follow the steps given below to setup collab notebook.
Upload Kaggle JSON Key
Upload Kaggle JSON Key
In order to connect our Kaggle account to collab, we need to download the Kaggle JSON key from the accounts section and upload it by running the following given below command:
#Run this cell and select the kaggle.json file downloadedfrom the Kaggle account settings pagefrom google.colab import filesfiles.upload()
2. Avoiding Errors if Any
By running the commands below we make sure that there are no errors and our Kaggle account is connected.
# Let's make sure the kaggle.json file is present.!ls -lha kaggle.json# Next, install the Kaggle API client.!pip install -q kaggle# The Kaggle API client expects this file to be in ~/.kaggle,# so move it there.!mkdir -p ~/.kaggle!cp kaggle.json ~/.kaggle/# This permissions change avoids a warning on Kaggle tool startup.!chmod 600 ~/.kaggle/kaggle.json
3. Downloading the Dataset
For this article, we will be using a dataset with the name ‘Disaster Tweets’, the commands are given below will search that dataset on Kaggle and we will select them for use.
!kaggle datasets download -d vstepanenko/disaster-tweets
We will now import some required libraries which will be useful for this article. Run the following commands to import the required dependencies.
import pandas as pdimport pandas_profiling as ppfrom nlp_profiler.core import apply_text_profiling
Now we need to unzip the dataset and create a dataframe of the columns we will be working on.
!unzip disaster-tweets.zipdf = pd.read_csv('tweets.csv')text_nlp = pd.DataFrame(df, columns=['text'])text_nlp.head()
In this step, we will apply text profiling to the text data which works on the data in order to extract useful information like Sentences count, Word Count, Spaces Count, Emoji Count, etc.
profile_data = apply_text_profiling(text_nlp, 'text')
After this let us call the describe function on this and analyze some properties of this dataset.
profile_data.describe()
profile_data.columns
So here we can clearly what all properties are extracted from the text data, in the image above you can clearly see all the columns which represent the analysis on different properties of the dataset.
Next we will visualize some properties of that are there in the profile data which is created using text profiling.
Sentiment Polarity Score
Sentiment Polarity Score
fig = px.histogram(profile_data, x="sentiment_polarity_score")fig.show()
2. Sentiment Polarity
fig = px.histogram(profile_data, x="sentiment_polarity")fig.show()
3. Sentiment Subjectivity
fig = px.histogram(profile_data, x="sentiment_subjectivity_summarised")fig.show()
4. Ease of Reading
fig = px.histogram(profile_data, x="ease_of_reading_score")fig.show()
Now let us look at some of the other properties.
Finding Correlated Properties
Finding Correlated Properties
# Finding the most and least correlated feature pairsdef most_correlated_pairs(dataframe, threshold=0.05):    corr_matrix = dataframe.corr()    indexes = corr_matrix.columns    pair_names = []    values = []    abs_values = []    for row_index in indexes:        for col_index in indexes:            if str(row_index) != str(col_index):                pair_name = f'{row_index} v/s {col_index}'                alt_pair_name = f'{col_index} v/s {row_index}'                if (pair_name not in pair_names) and (alt_pair_name not in pair_names):                    pair_names.append(pair_name)                    values.append(corr_matrix[row_index][col_index])                    abs_values.append(abs(corr_matrix[row_index][col_index]))correlation_pairs = pd.DataFrame({        'pair_name': pair_names,        'value': values,        'abs_value': abs_values    }).sort_values(by='abs_value', ascending=False)    return correlation_pairs[correlation_pairs.abs_value >= threshold]profiled_text_correalted_pairs_dataframe = most_correlated_pairs(profile_data, threshold=0.05)profiled_text_correalted_pairs_dataframe
Similarly we can create correlated features tree.
2. Correlation Tree
# Correlated feature trees (groups)def correlated_tree(dataframe, threshold=0.05):        corr_matrix = dataframe.corr()    indexes = corr_matrix.columns    nodes = {}    for row_index in indexes:        for col_index in indexes:            value = corr_matrix[row_index][col_index]            if (str(row_index) != str(col_index)) and (value > threshold):                value_as_str = f'{col_index} ({str(abs(round(value, 3)))})'                if row_index not in nodes:                    nodes[row_index] = []nodes[row_index].append(value_as_str)        return dict(sorted(nodes.items(), key=lambda item: item[0]))profiled_text_correalted_tree = correlated_tree(profile_data, threshold=0.07)for each_node in profiled_text_correalted_tree:    print(each_node)    for each in profiled_text_correalted_tree[each_node]:        print(f'└─ {each }')    print()
This is how you can explore different properties of textual datasets using NLP Profiler easily without any hassle.
Go ahead try this with different textual datasets, in case you find any difficulty you can post that in the response section.
This post is in collaboration with Piyush Ingale
Thanks for reading! If you want to get in touch with me, feel free to reach me on hmix13@gmail.com or my LinkedIn Profile. You can view my Github profile for different data science projects and packages tutorials. Also, feel free to explore my profile and read different articles I have written related to Data Science.