NLP Profiler: Profiling datasets with one or more text columns
By: Himanshu Sharma
Tags: Data Science, NLP, Data Analysis, Exploratory Data Analysis, Data Visualization

NLP Profiler: Profiling datasets with one or more text columns

Photo by Shahadat Rahman on Unsplash

Have you ever worked with a large text dataset, if you have then you must know how difficult it is to analyze text data, you need to use different functionalities of Natural Language Processing in order to get an idea of the data you are working with? But if I tell you that now don’t need to worry about performing a number of operations in order to analyze text data then it would be great right.

So let’s get started with NLP Profiler, which is a python tool/package/module used to create a summary of the data similar to that created by using describe() function of pandas for numerical data in a dataframe. It allows you to create a profile of textual datasets and analyze different properties of the text data. It generates high-level insights into the data you are using.

In this article, I will take you through different functionalities of NLP Profiler and I will show you how we can use that. We will be working on this using Google Collab.

Installing NLP Profiler

We will start by installing NLP Profiler by the command given below. As we are using Google Collab so you need to copy this command and run it in your collab notebook to install the NLP profiler.

!pip install nlp_profiler

Connecting Collab to Kaggle

In order to start exploring NLP Profiler let us start by connecting our collab notebook with Kaggle, this is important because we will directly load the dataset from Kaggle to the collab notebook. Follow the steps given below to setup collab notebook.

Upload Kaggle JSON Key

In order to connect our Kaggle account to collab, we need to download the Kaggle JSON key from the accounts section and upload it by running the following given below command:

#Run this cell and select the kaggle.json file downloaded

from the Kaggle account settings page

from google.colab import files

files.upload()

2. Avoiding Errors if Any

By running the commands below we make sure that there are no errors and our Kaggle account is connected.

# Let's make sure the kaggle.json file is present.

!ls -lha kaggle.json

# Next, install the Kaggle API client.

!pip install -q kaggle

# The Kaggle API client expects this file to be in ~/.kaggle,

# so move it there.

!mkdir -p ~/.kaggle

!cp kaggle.json ~/.kaggle/

# This permissions change avoids a warning on Kaggle tool startup.

!chmod 600 ~/.kaggle/kaggle.json

3. Downloading the Dataset

For this article, we will be using a dataset with the name ‘Disaster Tweets’, the commands are given below will search that dataset on Kaggle and we will select them for use.

!kaggle datasets download -d vstepanenko/disaster-tweets

Importing Required Libraries

We will now import some required libraries which will be useful for this article. Run the following commands to import the required dependencies.

import pandas as pd

import pandas_profiling as pp

from nlp_profiler.core import apply_text_profiling

Preparing the Dataset

Now we need to unzip the dataset and create a dataframe of the columns we will be working on.

!unzip disaster-tweets.zip df = pd.read_csv('tweets.csv')

text_nlp = pd.DataFrame(df, columns=['text'])

text_nlp.head()

Dataset(Source: By Author)

Applying Text Profiling

In this step, we will apply text profiling to the text data which works on the data in order to extract useful information like Sentences count, Word Count, Spaces Count, Emoji Count, etc.

profile_data = apply_text_profiling(text_nlp, 'text')

After this let us call the describe function on this and analyze some properties of this dataset.

profile_data.describe()

Statistical Properties(Source: By Author)

profile_data.columns

Columns in Profiling(Source: By Author)

So here we can clearly what all properties are extracted from the text data, in the image above you can clearly see all the columns which represent the analysis on different properties of the dataset.

Visualizing Text Properties

Next we will visualize some properties of that are there in the profile data which is created using text profiling.

Sentiment Polarity Score

fig = px.histogram(profile_data, x="sentiment_polarity_score")

fig.show()

SPE(Source: By Author)

2. Sentiment Polarity

fig = px.histogram(profile_data, x="sentiment_polarity")

fig.show()

Sentiment(Source: By Author)

3. Sentiment Subjectivity

fig = px.histogram(profile_data, x="sentiment_subjectivity_summarised")

fig.show()

Sentiment Subjectivity(Source: By Author)

4. Ease of Reading

fig = px.histogram(profile_data, x="ease_of_reading_score")

fig.show()

Ease of Reading(Source: By Author)

Some Other Properties

Now let us look at some of the other properties.

Finding Correlated Properties

# Finding the most and least correlated feature pairs

def most_correlated_pairs(dataframe, threshold=0.05):

corr_matrix = dataframe.corr()

indexes = corr_matrix.columns

pair_names = []

values = []

abs_values = []

for row_index in indexes:

for col_index in indexes:

if str(row_index) != str(col_index):

pair_name = f'{row_index} v/s {col_index}'

alt_pair_name = f'{col_index} v/s {row_index}'

if (pair_name not in pair_names) and (alt_pair_name not in pair_names):

pair_names.append(pair_name)

values.append(corr_matrix[row_index][col_index])

abs_values.append(abs(corr_matrix[row_index][col_index])) correlation_pairs = pd.DataFrame({

'pair_name': pair_names,

'value': values,

'abs_value': abs_values

}).sort_values(by='abs_value', ascending=False)

return correlation_pairs[correlation_pairs.abs_value >= threshold] profiled_text_correalted_pairs_dataframe = most_correlated_pairs(profile_data, threshold=0.05)

profiled_text_correalted_pairs_dataframe

Correlation(Source: By Author)

Similarly we can create correlated features tree.

2. Correlation Tree

# Correlated feature trees (groups)

def correlated_tree(dataframe, threshold=0.05):

corr_matrix = dataframe.corr()

indexes = corr_matrix.columns

nodes = {}

for row_index in indexes:

for col_index in indexes:

value = corr_matrix[row_index][col_index]

if (str(row_index) != str(col_index)) and (value > threshold):

value_as_str = f'{col_index} ({str(abs(round(value, 3)))})'

if row_index not in nodes:

nodes[row_index] = [] nodes[row_index].append(value_as_str)



return dict(sorted(nodes.items(), key=lambda item: item[0])) profiled_text_correalted_tree = correlated_tree(profile_data, threshold=0.07)

for each_node in profiled_text_correalted_tree:

print(each_node)

for each in profiled_text_correalted_tree[each_node]:

print(f'└─ {each }')

print()

Correlation Tree(Source: By Author)

This is how you can explore different properties of textual datasets using NLP Profiler easily without any hassle.

Go ahead try this with different textual datasets, in case you find any difficulty you can post that in the response section.

This post is in collaboration with Piyush Ingale

Before You Go

Thanks for reading! If you want to get in touch with me, feel free to reach me on hmix13@gmail.com or my LinkedIn Profile. You can view my Github profile for different data science projects and packages tutorials. Also, feel free to explore my profile and read different articles I have written related to Data Science.