Getting started with NLP using IBM Watson Studio
By: Aritro Mukherjee
Tags: NLP, Natural Language, Watson Nlu, Ibm Watson, Watson

Hola!!!

Welcome to the blog-post on Natural Language Processing by leveraging the capabilities of IBM Watson’s NLU(Natural Language Understanding) service.

Prerequisite: An open-mind

For newbies like me in the field of NLP, I feel images are more communicative in terms of story-telling. So, I have added screenshots and code-snippets wherever possible.

I’ll try to keep our learning experience as relevant to industry as possible.

To start using the start-of-the-art Watson Services, you need to first create an account on IBM Watson. Don’t worry about the costs incurred! I’ll try to stick to only free-tier services.

Today’s task:

We’ll be uploading a dataset into IBM Cloud Storage and do data transformation task on our notebook, while communicating with Watson’s APIs to get the sentiment of our reviews-dataset!

We’ll be using IBM Watson Studio for this post. Sign-up with Watson for free trial version for your testing purposes. After successfully creating your new account, you’ll be greeted by the following homepage.

Click on the “Create a project ” button to continue.

Now, we will create our first project by clicking on the New Project button.

You’ll need to provide a name for your notebook in the dialog-box, keep the Plan as Lite (free-tier). Once, done click on Confirm to continue.

Whenever you are selecting your plan of choice, read the features thoroughly.

Under the free-tier category, you’ll be able to make 30k service calls. We’ll see how to make use of such services under strict boundaries, to avoid incurring costs.

Once the new project is created, it should be listed under the NLU category.

Once, you go inside the project, you’ll find that there aren’t any credentials.

We’ll be needing credentials to make API calls to Watson’s NLU service, so let’s click on the New Credential button and keep a backup of the credentials in a notepad.

You’ll be asked to give a name , so go ahead and name your credentials!

Listed below are the various attributes of the service credential. We’ll be using “api_key” and the “url” endpoint for the current use-case.

Once our service is created, We will now build our NLP project.

Select the first option, to create an empty project.

In the dialog-box, you’ll be prompted to enter details, along-with a storage location. Scroll down to see how we create our Storage (IBM Cloud object storage).

Also, notice there’s an option to select a storage. Now, we need to specify a storage for our datasets. The setup is very simple, simply follow the screenshots below!

Just like NLU service, we’ll stick to the Lite (free-tier) plan for IBM Cloud object storage.

Update the Storage name in the dialog box and click on Confirm button.

Let’s now add our notebook, Click on “Add to project” button.

Select Notebook from the asset type. We’ll explore other assets in future iterations.

Select the environment for Python, including the CPU support you may require for your job.

We are done! Congrats…we’ve been successful to create our first notebook!

If you want to get a glimpse how the notebook looks like, click here.

Time to code!

It surely won’t end up this way!!

Once, your notebook is created, its going to look like this! A Blank Slate…

Default setup of Notebook

Data Ingestion

The very first component in an ML project is the dataset. You can download it here.

This dataset comprises of customer reviews in an e-commerce portal. Each of these reviews has been split up into multiple snippets, to allow us analyse its sentiment much better at a granular level.

Confused? Let’s see an example:

Review:

This shirt is very flattering to all due to the adjustable front tie. it is the perfect length to wear with leggings and it is sleeveless so it pairs well with any cardigan. love this shirt!!!

Sentence 1:

This shirt is very flattering to all due to the adjustable front tie.

Sentence 2:

and it is sleeveless so it pairs well with any cardigan.

Sentence 3:

love this shirt!!

I hope you’ve downloaded the dataset by now. Let’s quickly import it!

There’s a wierd icon on the top-right corner of the menu-bar. Refer to the snapshot below. You can either drag-&-drop or simply browse from local system. The dataset will be uploaded to IBM Cloud object storage.

You’ll always be able to view all your datasets under the “Data assets” category as shown below:

In order to read this dataset in our notebook, what if I said that you need not write a single code?

Enjoy the beauty of Watson Notebook!!!

Simply click on the “Insert pandas DataFrame” under the dataset, to automatically populate your notebook and creating a dataframe!

You’ll see something like this.

df_data_1 is your default dataframe.

To view, how the dataset looks like simply run df_data_1.head()

In order to analyse sentiments for each sentence, we’ll be using data from “sentences” column.

Data Transformation

In a real-world scenario, this layer would comprise of multiple action-items, which we’ll discuss in our future iterations.

In this tutorial, we’ll consider first 10 records from “sentences” column and store it into a new dataframe. Subsequently, we’ll filter out the important columns once we invoke Watson’s NLU API.

Execute the following code to copy first 10 records and store it in df1:

Cognitive Analysis

We can infer properties from the customer-reviews using the following options provided by Watson’s NLU as:

Features, EntitiesOptions, KeywordsOptions,SentimentOptions.

In this current tutorial, we’ll incorporate EntitiesOptions and SentimentOptions

We’ll be importing all the necessary packages and include the API key to facilitate NLU-based analytics.

As I have already said, in today’s use-case we’ll find the sentiment score for a review and also the relevant keywords in that review. We’ll execute this operation for our complete dataset.

The example below will bring more clarity:

Sentence:

This would definitely be a true midi on someone who is truly petite.

Corresponding Sentiment:

{"sentiment": {"document": {"score": 0.89721,"label":"positive"}}}

Corresponding Keyword/s:

{“keywords”:[{ “text”:“true midi”,“relevance”:0.5,“count”: 1}]}

As can be seen here, the output payload from NLU API service calls comprise of JSON data.

Since, we are using the Lite-tier version, be cautious about the number of API calls you make.

Thus, we have created a method to compute the sentiment and keywords for a review in a single service call.

Hence, we would be extracting our required attributes along-with their corresponding scores from this JSON data.

We’ll now apply the API service calls row-wise for our dataset.

Let’s see how the resultant data-frame looks like. Click here to download notebook.

Congrats!! You’ve learnt how to implement Watson services for concurring NLP analysis.

I’m glad you stayed with me for so long.. Here’s a perk for you!

J OB SCHEDULING

In many instances, while working on an industrial project, you may be asked to run a job at repetitive intervals, whether it is for data ingestion, data cleaning or data transformation.

A utomating the boring task!

That’s really BORING!!!

So, let’s check out this trick! and I promise its super easy to setup!

On the top right corner of the notebook, there’s an option to “create a job”.

We’ll now be redirected to a new dashboard specifically dedicated for job creation.

You’ll get a provision for selecting an environment. Select the python version and other attributes to get started.

Once we’ve named our new job, it’s time to click on the slider on the right side of the dashboard. We can now schedule to execute the notebook repetitively by specifying the start date and setting the time.

Once, the setup is complete, we’ll be redirected to the dashboard as below.

All the job runs, once executed successfully can be viewed under “Runs” category. You’ll be able to see the time taken for the model to execute.

Voila!!!! You’ve not only created a notebook, but also scheduled it to run at regular intervals! How cool is that!