fabiochiu/medium-articles huggingface dataset:
VUCA unpacked (3)—Perspective coordination
By: Theo Dawson
Tags: Perspectives, Perspective Coordination, Decision Making Tools, Leadership Development, Micro Skills

Perspective-coordination skills build upon and extend collaborative capacity skills. They are essential for constructing knowledge, resolving conflicts, solving problems, and making decisions. In other words, they are required throughout our lives for learning, working, living, and playing with others.

What is perspective coordination?

Perspective coordination is a dynamic set of skills that supports human interactions by fostering mutual respect, nurturing creativity, expanding our minds, generating and developing ideas, leveraging conflict, and supporting healthy relationships. Robust collaborative capacity skills are a prerequisite for the development of perspective coordination skills, but you will also find that practices for building perspective coordination skills build collaborative capacity skills. Wheels within wheels.*

As shown in the skill map below, we divide the perspective-coordination mega-skill into three macro-skills — (1) identifying relevant perspectives, (2) supporting productive interactions, and (3) integrating perspectives. Each macro-skill is further broken down into mini-skills (in bold type), then micro-skills.

Identifying relevant perspectives: When it comes to perspective coordination, skills for identifying relevant perspectives come first. The primary mini-skills in this group include: consult stakeholders, evaluate sources, and consult trusted sources. Whatever your reason for applying perspective coordination skills—planning a family holiday, resolving a national crisis, or engaging in collaborative learning or problem-solving, you need to be able to identify and evaluate relevant perspectives. Choosing wisely ensures that the perspectives that matter are well-represented.

Supporting productive conversations: Ideally, any time we are involved in a conversation we’re exercising skills that help make that conversation more productive. This is especially important when the conversation is part of a learning, problem-solving, negotiation, or decision-making process. Skills that support productive conversations fall into two primary mini-skill categories: cultivate respectful interactions and facilitate generative interactions.

Integrating perspectives: Productive conversations tend to produce a great deal of information. To make the most of this bounty, we need skills for organizing information and making robust connections between the needs, values, facts, and ideas that have emerged. We also need skills for identifying and working with the tensions that arise during productive conversations, along with skills for uncovering common ground and employing conflict resolution techniques. The primary mini-skill categories for integrating perspectives include: organize information, work with tensions, employ conflict-resolution tools, seek common ground, and make robust connections.
The Struggle to be Supportive of a Culture
By: Jeff Ehren
Tags: Culture, Empathy, Social Media, Swastika, Perspective

The Struggle to be Supportive of a Culture

Photo by Arie Wubben on Unsplash

This past week, the Anne Frank Human Rights Memorial in the state of Idaho was vandalized. The symbol used by the Nazis — known to many as the “swastika” — was printed on stickers along with the phrase, “We Are Everywhere.”

The incident was reported by the Idaho Press, the Jewish Herald-Voice, NBC, NPR, Brazil 247, among others. Whether the term “Nazi propaganda” or “swastika” was used, the messaging was clear to the so-called “Western World.”

But is it a swastika?

Yes and no.

The symbol has been around for thousands of years and used by different cultures. The symbol therefore means something different to different people. This idea can be understood in the lecture scene from the popular movie “The Da Vinci Code.”

Following the vandalism incident, social media influencer and Jewish woman Roxy Striar used the word “swastika” during her condemnation of the act. And then two missteps happened due to cultural differences and a lack of combining empathy with a teachable moment.

First, instead of showing empathy, some on social media preferred to point out that “swastika” is the wrong term. This is technically accurate from historical and certain cultural perspectives. However, it doesn’t change the fact that the use of the symbol was targeted at a specific culture — Jewish people.

Secondly, in reaction to the seemingly outright dismissal of this targeted vandalism, Striar made a statement that everyone should just stop using this symbol. This obviously is an overreach, and it inflamed the situation. However, it would be a legitimate request for Striar to say, “This symbol offends me and I don’t want to see it anywhere.”

Social media can be a wonderful place to learn about other cultures, but it can also give power to racism, bigotry, classism, xenophobia and other forms of hatred. It can be difficult to say, “I’m sorry this happened. I’m sorry you are hurt. When you have a moment to breathe, I’d like to talk to you some more about the context of the situation.”

It is wonderful to see Kamala Harris become representative of so many underrepresented cultures in the United States. At the same time, there has been a resurgence of Nazis and antisemitism.

We can celebrate the good while condemning evil. We can “code switch” when communicating with different groups to send the same message. We can enjoy something seemingly wonderful in our own lives while understanding that it may hurt others in theirs.

Technology is wonderful, but it doesn’t always allow us to sit with another human being, break bread, and have a deep, civil conversation about culture.
Tao of a Flutterby
By: Katrina Dominique
Tags: Perspective, Kids Stories, Encouragement, Poetry Writing

Tao of a Flutterby

Just like the little caterpillar whose



Humble beginnings, and life’s unpaid dues;



Are hard and are challenging, its purpose unclear:



It stays in the moment…the present its sphere.

It sticks to the plan, keeps plodding along.



Eating and growing, his fortitude strong.

He doesn’t worry about other bugs.



Doesn’t need our approval, our friendship, our hugs.



He’s happy just being, just living, just fine!

‘Til one day it’s over. It’s done! He’s resigned,



To his fate, his demise…It’s the end at long last!



To becoming a memory, an echo, the past.

Not once did he scream out, bemoan, or complain…



His life, or his fate, his choices disdain.



He was what he was, had done what he’d done!



Just a lone cat-er-pillar, all fat, his bed spun!







So he sealed himself off, and like that (snap)he was dead!..………………………………………………………………………………………………………………………………………………………………………………………………………………………. Now, I know what you’re thinking, (know what goes through your head. )

The answers your lacking…



Confusion you’re masking…

But, I’ll let you in on a little known fact…



Life’s struggles, its purpose, rewards you have stacked!

It’s really quite simple, in fact a no- brainer!



The answers you seek, just couldn’t be plainer.

Remember our friend? The tiny wee insect?



Persistent, relentless, his focus correct?

From the very beginning our friend was shortsighted!



He was not a crawler, his destiny flighted!!!!

A creature of Beauty, A marvel, Amazing!



His time from before? Was simply him phasing!

What we see with our eyes, and think we know now…



Is simply perspective; (or scope) of our Tao…







So whenever you think that the battles been lost,



That your time, or your efforts, are too dear a cost:

Just remember our friend. His phases. Life’s twists!



Be mindfully present. Keep moving. Exist!

For one day you’ll look back at the time here you’ve spent,



And emerge from the journey changed 100%!



















By Katrina Marie Dominique
Social Media Can Save Us
By: Mike Vardy
Tags: Thinking, Covid 19, Problem Solving, Social Media, Perspective

Photo by John Torcasio on Unsplash

I’m getting tired of hearing that social media is a massive problem. Anything can be a problem when you look at it from one vantage point over another.

In the city where I live, there are people that believe cars are a problem and that we should have more bike lanes that prioritize cycling over driving. I’m not suggesting that they don’t have a good point, but they often look at this as a problem because of their vantage point. Seniors don’t ride bikes nearly as much as others and we do have plenty of seniors that live here. That means that having more cycling in place of cars could be a problem for them instead. First responders are having to take longer routes to do their job because of roads that have been closed off to motor vehicle traffic. That’s another problem — not just for them but possibly for those they’re trying to serve.

It’s all about perspective.

Social media falls into the same arena. There are plenty of bad things about social media, particularly if you spend too much time on the different social media platforms available to us. But without social media, I wouldn’t have friends all over the world. I wouldn’t be able to connect with people in the midst of a global pandemic nearly as easily. From that perspective, I believe that social media can save us. It helps us stay connected at a time where connection in person is hard (if not impossible) to do.

Social media is only really a problem if you let it become one. And that happens when we don’t step back and figure out how we want to navigate it.

I suppose the same could be said for the cycling situation here in my city. Both bikes and cars need to navigate the city streets together, not forsaking either in the process. What if the decision-makers and policy-makers in my city took a step back and looked at things from a different perspective? What problems could be solved better? What problems would be effectively dismissed at not really being problems at all?

When I worked for Costco, one of my managers said something that has stuck with me ever since: If you come to me with a problem, then come with a solution to that problem. I think that’s something all of us should do when investing time in social media.

Actually, I think that’s something we should all do for a lot of things.
What Would This Look Like If Life Were Incredible?
By: Kaz Kawasaki
Tags: Motivation, Perspective, Inspiration

Too often we get caught up in our own worlds. We end up just mindlessly doing things. Day after day. Our evening turn into mornings turn into going into work turn into the weekend and the whole things just keeps going round and around again.

Then we look up once in a while. And we suddenly awaken. And we ask ourselves, “what am I doing?” The past days, hours, maybe years have passed you by in this fog of routine. You’re used to doing things because that’s where you settled. But what would happen if you actually made your life amazing? You worked out more. You ate better. You treated your job like you were meant to be good at it. What if you decided to make your life awesome instead of treating yourself like a passerby in the movie of your own life?

Even when applied to the everyday drudgery of daily chores, your world transforms. What if grocery shopping with your wife was incredible? What if you made an effort to make every car ride with your kids an awesome and memorable experience? What if you strove to live every morning like it’s on purpose, and not just trudging through the everyday-ness of life?

Because trust me, there is no end of situations that will take you down if you let them. Look around and you’ll find no end of people who walk and talk and move like their life is one giant pit full of sinking sand and their soul is stuck in it.

Make no mistake, I’m not calling stupid optimism rooted in willing ignorance. But instead I’m telling you that you can actively transform the interactions around you to juice them for all the potential in them.
Love
Tags: Love, Perspective, Bahasa, Of

“If you have a garden and a library, you have everything you need.”
Guide to use character BERT for sentiment analysis using Pytorch
By: Siddhanth Hegde
Tags: Naturallanguageprocessing, Data Science, Deep Learning, Bert, Neural Networks

Photo by Cookie the Pom on Unsplash

This guide mainly focuses on using character BERT on a python notebook rather than a script. This guide does not explain the how it works. Click here to know the working of BERT and here to know about character BERT. This idea was originally proposed by Hicham EL BOUKKOURI

Comparison of BERT embeddings and character BERT embeddings [1]

Original github repository of character BERT can be found here.

For local notebook users create a conda environment.

conda create python=3.8 --name= environment_name

If not activated, activate environment using,

conda activate environment_name

Latest version of transformers might not work. It is better to use the version compatible with the repository.

pip install transformers==3.3.1

Clone the github repository and change the directory to the repository

git clone https://github.com/helboukkouri/character-bert.git %cd character-bert

Download a variant of the model

python download.py --model='medical_character_bert'

To download all the models:

python download.py --model='all'

Import necessary stuffs

Instantiate a base BERT model. I have used bert-base-cased. Other variants can also be used such as ‘bert-multilingual-cased’ etc.

Initiate the tokenizer and convert it to cuda if gpu is available

Load the data, convert sentiment column into label encoding and tokenize the texts. The indexer converts each token into a list of character indices.

Split the data and then create a dataloader

Define the loss. Decay is added and is totally optional

We can finally start training now…Yayy!!

Like this character BERT can be used for other tasks like machine translation, text summarizers etc. Even though character BERT did not perform as well as the normal BERT because it was not trained on that huge dataset. For some tasks character BERT produced more robust results than the normal BERT.

Thank you for reading!!! Complete code for this is available here
What is NLP — Natural Language Processing
By: Hhussain Wali
Tags: NLP, What Is Nlp, Naturallanguageprocessing

Few Examples of Applications of NLP

Livox App: Helps people with disabilities to communicate. It was developed by the father of a disabled girl to understand his daughter.

SingAll: Converts sign language into text, it’s helping individuals who are deaf communicate with everyone else who don’t know sign language.

Google Translate: For communication, understanding manuals, books, catalogs written in foreign language. Used by more than 500 million people to learn and understand every other language in the world

Aircraft Maintenance: NLP is being used in the aviation industry for aircraft maintenance. It helps mechanics synthesize useful information from enormous aircraft manuals. It also helps find meaning in the descriptions of problems reported verbally or handwritten from pilots and other humans.

Predictive Police Work: NLP can help identify individuals’ motives in crimes. Studying the behavior of suspicious individuals beforehand and predicting crimes.

Virtual Assistants: Alexa, Siri, Google Voice search, Cortana all are powered by NLP

Q&A Systems: Question and Answering systems have been employing NLP since 1993

Automatic Exam Marking Systems: Moodle, ETS e-rater, assessment21 the modern way of marking exams

News Analysis for Market Intelligence: Aggolo systematically reads, organizes, and synthesizes news and other textual information, generating configurable summaries, easily shareable with colleagues or clients.
Spoken Language Understanding : On Building Spoken Dialogue Systems
By: Akshat Gupta
Tags: Dialog System, Naturallanguageprocessing, Speech Recognition, Deep Learning

Building Spoken Dialogue Systems for Low Resource Languages

Although Spoken Dialogue Systems like Siri and Alexa seem to be widely available and perform decently well on most tasks they are build for, such systems are not available in most languages around the world. Bottleneck is usually the absence of a robust speech to text system for low resource languages.

Our group at CMU is working on building Spoken Dialogue Systems for low resource languages. The general idea of our work is similar to reasons behind building an E2E-SLU system - do we really need to build full scale and specific STT and NLU modules to build spoken dialogue systems?

We represent user audio, which could be in any language, into a universal set of symbols. Using this, we avoid building specific and data intensive speech-to-text systems that are language specific. These universal set of symbols are a set of (nearly) universal phones (generated using Allosaurus [4]), which are common to (almost) all languages. We then use these universal symbolic transcripts of audio to do downstream NLU tasks. A block diagram representing such a system is shown below.

Block diagram of our proposed SLU system for low resource languages as shown in [5]

The complexity of creating full-blown STT systems, which needs to be built separately for different languages due to differences in written forms of the language is reduced to representing speech in terms of phones. On top of that, this technique also generalizes to unwritten languages. These reduced representations of speech are then used to perform downstream tasks such intent recognition, slot filing etc. In [5], we do this for Indian and European languages and show that our methods attain deployable accuracies for both language families.
Spam detection in 15 lines of code
By: Edward Wang
Tags: Naturallanguageprocessing, Sklearn, Machine Learning, Artificial Intelligence, NLP

DIY machine learning model in 3 minutes

Machine learning has always been seen as a black box, with its root in calculus, linear algebra, and statistics. In reality, it’s much more compact and easier than most algorithms out there. Here’s how to build a spam detection algorithm in 15 lines of code. Follow alongside with me or just skim through!

Import necessary packages

import pandas as pd

from sklearn.model_selection import train_test_split

from sklearn.feature_extraction.text import CountVectorizer

from sklearn import svm

import pandas as pd imports pandas a data cleaning and analysis package. “as pd” allows us to call pandas methods as “pd” and not “pandas”.

from sklearn.model_selection import train_test_split a package that allows us to split our dataset into training sets, where we teach our model, and a testing dataset where we test how well our model does.

from sklearn.feature_extraction.text import CountVectorizer a package that tokenizes and counts the occurrences of words in an email. Tokenizes just means gives each word a random number for easier manipulation.

from sklearn import svm import svm a support vector machine that does trains the model with the data. If there is a secret sauce behind machine learning this is it.

Data analysis

spam = pd.read_csv('spam.csv')

spam = pd.read_csv('spam.csv') turns the CSV file ‘spam.csv’ into a data frame and saves it to spam. A data frame is just tabular data that python can interpret, no changes to the dataset are made. Find the data here.

Split Training and Test Dataset

x = spam['EmailText']

y = spam["Label"]

x_train, x_test,y_train, y_test = train_test_split(x,y,test_size = 0.2)

x = spam['EmailText']: assigns the column “EmailText” from spam to x. This is the data we put through the model. You can see a screenshot of the raw dataset below.

y = spam["Label"]: assigns the column “Label” from spam to y. This tells the model to correct the answer, and how it does. You can see a screenshot of the raw dataset below.

x_train, x_test,y_train, y_test = train_test_split(x,y,test_size = 0.2): divides columns x, and y into x_train for training inputs, y_train for training labels, and x_test for testing inputs, and y_test for testing labels. test_size=0.2 sets the testing set to 20% of x and y.
Dutch GPT2: Autoregressive Language Modelling on a budget
By: Thomas Dehaene
Tags: Naturallanguageprocessing, Text Generation, Machine Learning, Artificial Intelligence, NLP

In the following deepdives, we only retain key snippets of code in this post, instead of plowing through each line of code 🚜. Don’t let us keep you from your beloved plowing though, right this way to the full code 👉 link 👈!

Step 1: Training a Dutch tokenizer

One important part of a language model is the tokenizer, which essentially converts the input sequence into tokens that the embedding layer of the model understands and embeds in a meaningful fashion.

For this, we initialize a fresh ByteLevelBPE tokenizer and train it on our Dutch OSCAR corpus.

Things to note:

👆 we use the same vocab size as the English GPT2 tokenizer

👆 we save it and re-initialize it as a GPT2TokenizerFast object

Step 2: Fastai tokenizer + initial embeddings by comparing both tokenizers

Further down the road, we need two things: a fastai tokenizer for the dataloader, and an updated Dutch embedding matrix.

For the first, we can simply convert it using a one-liner (see below). This tokenizer is used further down in the dataloader.

Secondly, we will update the embedding matrix of the English GPT2 model to only keep the common tokens between the Dutch and English vocabulary. The rest will be set to the mean embedding weight for further finetuning during the training phase.

Step 3: Creating a fastai dataloader

We have data…

The data needs to get loaded…

So we need a fastai dataloader 😉

This also re-uses the Dutch tokenizer we made earlier!

Step 4: splitting the model for gradual unfreezing

As is good practice in finetuning large models, we will do so in a gradual unfreezing fashion!

Each subsequent run will thus also have more parameters to train on.

We divide the decoder (12 layers for GPT2 small, 24 for GPT2 large, etc.) in 3 equal parts.

As such, we finetune the following layers per run:

The LM head + embedding LM head + embedding + decoder block 3 LM head + embedding + decoder block 2

The underlying (though difficult to theoretically or experimentally prove) idea of this is that the earlier decoder blocks would focus more on semantics and structure, while the later layers focus more on the actual dialect (English, Dutch, etc.)

This split is provided to the final Learner along with:

the dataloader created earlier

the base model to start from

a loss function

Step 5: training time

Now for the easy part! We gradually unfreeze the aforementioned groups one by one, and each time fit a single cycle (one run of the entire dataloader).

We didn’t really do much hyperparameter tuning on the learning rate, as our budget and timing for training this beast was limited.

In the end, we save the model, the fruit of our hard labor, some place nice and safe!
Text Recognition Workflows | How to Use Clarifai’s Advanced Text Workflow
By: Thomas Molfetto
Tags: Naturallanguageprocessing, Deep Learning, Machine Learning, Optical Character Recogn, Computer Vision

By Jeff Toffoli

We are retiring our OCR model and have replaced it with a new and improved Visual Text Recognition (VTR) workflow. The new workflow separates the components of text recognition so that you can work with detection, cropping and text recognition as individual components in your projects.

The VTR workflow is already configured for you when creating an app in Portal. Just select “Visual-Text-Recognition” when selecting the base workflow for your app.

You can inspect the individual models in the workflow by clicking the “Copy to new Workflow” button in the “App Workflows” section of the App overview page.

Click here to find out more about making workflow predict calls.

The VTR Workflow Post Workflow Results Request in our API

The new VTR workflow uses a call to the workflow endpoint in our API (as opposed to the OCR model where you were making a single “model predict” call).

We are retiring our OCR model and have replaced it with a new and improved Visual Text Recognition (VTR) workflow.
CrossNER: Evaluating Cross-Domain Named Entity Recognition
By: Zihan Liu
Tags: Dataset, Naturallanguageprocessing, Cross Domain, Named Entity Recognition, Multi Domain

Motivation of CrossNER

We Need Cross-Domain NER Models

Named entity recognition (NER) is a key component in text processing and information extraction. Contemporary NER systems rely on numerous training samples and a well-trained NER model could fail to generalize to a new domain due to the domain discrepancy. However, collecting large amounts of data samples is expensive and time-consuming. Hence, it is essential to build cross-domain NER models that possess transferability to quickly adapt to a target domain by using only a few training samples.

We Need a Better Benchmark to Evaluate Cross-Domain NER Models

There are two drawbacks when utilizing existing datasets for cross-domain NER evaluation. First, most targetdomains are either close to the source domain or not narrowed down to a specific topic or domain. Second, the entity categories for the target domains are limited. To this end, we collect a cross-domain NER dataset (CrossNER), a fully-labeled collection of NER data spanning over five diverse domains (politics, natural science, music, literature and artificial intelligence) with specialized entity categories for different domains. Additionally, we also provide a domain-related corpus since using it to continue pre-training language models (domain-adaptive pre-training (DAPT)) is effective for the domain adaptation (Gururangan et al. 2020).

The CrossNER Dataset

To collect CrossNER, we first construct five unlabeled domain-specific (politics, natural science, music, literature and AI) corpora from Wikipedia. Then, we extract sentences from these corpora for annotating named entities. We consider Reuters News from CoNLL2003 as the source domain data.

Data Statistics and Entity Category
Thai NLP — with no spaces. The first step in an NLP pipeline is…
By: Nick Doiron
Tags: NLP, Naturallanguageprocessing

One of the first steps in an NLP pipeline is dividing raw text into words or word-pieces, known as tokens. But what if you don’t have spaces to divide sentences into words?

Background

Sample Thai text from Wikipedia

People do write some spaces in Thai text, as you can see above, but they aren’t universal as they are in English. There is also no set punctuation to end a Thai sentence. This can cause confusion, or poetry, but humans are good at separating them in context. The difficult part, then, is getting computers to pick up on that context.

When I first heard about this text-parsing problem in early 2016 at the Asia Foundation, the best solution was to find a rules-based segmentation tool such as LibThai from the Linux Foundation.

In summer 2016, Wannaphong Phatthiyaphaibun created the first release of PyThaiNLP, a library inspired by NLTK.

As we enter the deep learning phase of NLP, a new generation of libraries have emerged using neural networks. Pattarawat Chormai (et. al)’s 2019 paper uses a benchmark to compare AttaCut (based on PyTorch), PyThaiNLP, DeepCut, and Sertis’s library (these last two use TensorFlow).

After tokenization, models

I found a few examples of recent Thai NLP models. There is a modern word2vec-type model, and a FastText release. mBERT has Thai language, but only in one version (cased).

I was curious how mBERT tokenizes Thai sentences:

tokenizing a Wikipedia article

I was surprised to see that mBERT dissolves Thai into almost character-level embeddings. The exceptions are a handful of common words such as ‘born in’, ‘Thailand’, ‘is’. These tiny tokens cannot be placed into meaningful vectors to for example, identify positive vs. negative, formal vs. informal, sports and government topics. This convinced me that a proper {segmenting → tokenizer → transformer} pipeline could improve Thai NLP, as it has in Arabic.

This repo is a Thai-specific BERT model created in 2018 based on tokens derived from {Wikipedia text → sentence segments → matching word-piece vocabulary → BERT}. While I reached out to the original developer, I decided to adapt their TensorFlow model checkpoint to HuggingFace’s transformers library.

The model is now available at huggingface.co/monsoon-nlp/bert-base-thai — unfortunately I have yet to prove an improvement over mBERT

2021 Update

While researching this article, I learned about https://airesearch.in.th/, a government-funded research center. In early 2021 they released several high quality Thai language models on https://huggingface.co/airesearch

Pre-print: https://arxiv.org/abs/2101.09635
Islamophobia and Natural Language Processing update (December 14, 2020)
By: Ted Pedersen
Tags: Islamophobia, Naturallanguageprocessing

I had the pleasure of giving an invited talk at the Mulims in Machine Learning Affinity Group Workshop at NeurlPS 2020, which just concluded.

My talk is 25 minutes long and provides an update on our Islamophobia and Natural Language Processing project. It is freely available via SlidesLive.

2019 updates:

July (project kickoff)

August (background reading, Ilhan Omar, Minnesota)

December (background reading, Genocide)

2020 Updates

May (annotation scheme)

August (data statement)

November (invited talk at UMass-Lowell)

Please stay in touch!
Decoding the Covid-19 tweets using NLP and Graph Database
By: Shyam Pratap Singh
Tags: Naturallanguageprocessing, Twitter, Graph, Neo4j, Vaccines

Analysis Flow :

Data processing flow. Image source Author

Data source: I have taken the Twitter data set ( corona_tweets_268.csv: 3,190,245 tweets (December 11, 2020, 08:00 AM — December 12, 2020 10:10 AM) related to corona from IEEE Dataport.

Twitter doesn’t allow to store/process entire user posts so once you gather Tweets ID, you have to use tools like Hydrator to extract post information from Tweets ID.

Once you get the Twitter post information in JSONL file, you can then convert to JSON format, export the file to any programming language to do data cleaning, data massage, and then import the file to neo4j. This is one of the tedious tasks.

Note: for sake of simplicity, you can download the twitter.json which contains 5000 tweets from the below GitHub link mentioned in the reference.

Core Tools and Libraries:

We are going to use the Neo4j database where we will store tweets and do further analysis.

Plugins to be installed in neo4j:

APOC pluggin

Graph data science pluggin

Graphaware: Graphaware has lots of libraries that offer NLP capability to Neo4j so we will use these libraries

List of libraries provided by graph aware:

neo4j-framework

neo4j-nlp

neo4j-nlp-stanfordnlp stanford-english-corenlp ( Language Model) This is language model file, need to be downloaded separately

Set up is one of the most difficult parts due to many dependent libraries and versions.

Follow the Installation step strictly in order as mentioned in guidelines to set up Graphaware libraries in neo4j.

Due to a lack of proper documentation, you may be lost in finding the right libraries and version. So here is one document I prepared to help you out in finding libraries and their appropriate versions.

I am using the Bloom tool for visualization which is optional.

At the end of set up your neo4j.conf file should look like this :

# nlp settings

apoc.import.file.enabled=true

dbms.unmanaged_extension_classes=com.graphaware.server=/graphaware

com.graphaware.runtime.enabled=true

com.graphaware.module.NLP.2=com.graphaware.nlp.module.NLPBootstrapper

dbms.security.procedures.whitelist=ga.nlp.*,gds.*,apoc.*

Plugins should have these jar files

import should have the twitter.json file which you have created earlier or you can download from my GitHub link.

Text Analysis :

We are going to follow the below step for text analysis:
Sentiment Analysis of a Youtube Video
By: Amit Ranjan
Tags: Machine Learning, Data Science, Naturallanguageprocessing, Sentiment Analysis

Selenium

Step 1: Install Necessary Libraries

For selenium we need to install it first in Google Colab. Below are the codes for installation.

It would take some time to install based on your internet speed.

Step 2: Import Necessary Libraries

After installing selenium we need to import all the libraries for scrapping the websites.

Step 3: Setup ChromeDriver and Scrap the Comments

We have taken an authors list to store all the comments made by youtube users.

Then, we configure the ChromeDriver for scrapping.

If you go on comment section of youtube you will find that the youtube comment section dynamically loads when you scroll down. For gathering at least 100 comments I have iterated for loop 6 times. Also, I have taken timer to stop for 5 sec to load the page. If you have slow internet connection make sure to increase the time so that page loads successfully and you don’t ran into exceptions.

Now comes the best part. We start scrapping all the comments and their author details.

We got uncleaned data and we have to clean it as it contains some of the extra details like when did the user commented and how many reply it got at the end.

In first comment, “

11 hours ago

” after username (Aspen Gaming) and “

26

REPLY” at the end of the comment.

We want our data to be like this,

where we have a user and his comment.

For removal of those extra details we need to clean it and then put it into data frame.

Step 4: Clean and Put data into DataFrame

In line 7, we are removing time of the comments after username like

11 hours ago

.

In line 8, we are removing

REPLY at the end of the comment.

In line 9, we are removing

followed by number of replies to that comment.

When a user edit their comments, youtube shows the username followed by (edited). This we have to remove from our username. This is what line 10- 12 is doing!

Lastly, we put author name as row name and their comment in column inside a Data frame.

Our final data after scrapping the website through selenium looks like:

Final DataFrame after scrapping

What are the problems we found here?

We had to do some cleansing before storing it into Data Frame.

These might not work if youtube changes its webpage design.

This is why I switched to Google API.
Exploring hidden data in Apple’s earnings call with text mining techniques
Tags: Naturallanguageprocessing, Data Science, Business, Finance, Apple

After generating the bigram network in R, I circled a few groups of words that I think could tell us something about Apple’s business model or strategy.

One theme emerged from three groups of words in the bigram network. Starting from the left, the “cloud” word-group; the “subscriptions” word-group at the bottom; and the “apple” word-group on the far right tells us that Apple’s future is beyond its smartphones or computers, and increasingly towards providing services value to its customers. Among the many services provided by the company, ranging from Apple TV, Apple Music, Apple care, etc., cloud service might be the company’s core service.

The “iPhone’ word-group also has the word “revenue” within, which may suggest that iPhone is one of its most profitable products. The “supply chain” word-group is linked with the word “constraint” and “constraints”, which may be the reason why the company is seeking to alter its operation by cutting costs. It is unlikely that “cost-savings” has restrained the supply chain as company representatives are unlikely to share bad decision moves in earnings calls.

POS tagging — Assigning parts of speech to each word

In this section, I used the udpipe package in R to tag the words in the transcripts into verbs, nouns, adverbs, adjectives, and so on. The most occurring nouns were similar to the result from the frequency analysis, hence I did not include them in this article. But, the most occurring proper noun bar chart provided more insights, as shown below:

Image by author: Most occurring proper nouns in Apple’s FY 2020 earnings calls

In the earnings calls, a number of countries were mentioned. The most mentioned country is “China”. Then I used the findAssoc function to find out how these two words correlate with any other words in the extracted transcripts.

Scores range from 0 to 1. A score of 1 means that two words always appear together, while a score approaching 0 means the terms rarely appear in the same document. Minimum correlation values are often relatively low because of word diversity.
Natural Language Processing
By: Amit Ranjan
Tags: Machine Learning, Artificial Intelligence, Naturallanguageprocessing, Data Science

Natural Language Processing

So if I visually express this it will be something like NLP is a baby product of all the three fields.

Let’s see why NLP is considered as subfield of all the three major fields?

Computer Science -> Programming

Artificial Intelligence -> You will apply Machine Learning Models

Human Language -> Text or Voice.

Does this explain NLP? I guess no! One question which pops up in my mind is that what is the purpose of the NLP?

The purpose of the Natural Language Processing is to read, decipher into machine language, understand and make sense of the human languages. Basically it is the way to teach the machine to understand what is given in spoken or written word.

Remember few years back we have to type everything into Google search to get effective results. Now we just use Google Voice Assistant or Siri ( if you are Apple user ). What changed?

Well it’s not so hard to guess! Google developed apps like Google Translate and Voice Assistant where we dictate something to our Android device that captures our voices in audio files, convert it into text, process the text’s data and search it over internet or convert them back to audio and present it to user.
Start your NLP journey with this Periodic Table of 80+ NLP tasks
By: Rob Van Zoest
Tags: Naturallanguageprocessing, Programming, Data Science, AI, NLP

Russian chemist Dmitri Mendeleev published the first Periodic Table in 1869. Now it’s time for the NLP tasks to be organized in the Periodic Table style!

The variation and structure of NLP tasks is endless. Still, you can think about building NLP Pipelines based on standard NLP tasks and dividing them into groups. But what do these tasks entail?
Text Classification Using a Transformer-Based Model
By: The Nyu Center For Social Media
Tags: Naturallanguageprocessing, Text Classification, Data Science, Research, Open Source

In this post, we show how to classify textual data using SMaBERTa — a wrapper for a stable version of RoBERTa language models. The code we provide was adapted from version 0.6 of simpletransformers. It uses the Simple Transformers library, which is built on top of the Transformers library by Hugging Face.

To install the package for this tutorial, run:

pip install smaberta

You can download and follow along via this notebook.
Automation in summary generation
By: Amish Gadhia
Tags: Naturallanguageprocessing, Python, Summarization, Python Programming

In this article we will explain how our summarization algorithm works, to create a summary of the text or an audio file.

Automatic summarization is the process of shortening a set of data computationally, to create a subset that represents the most important or relevant information within the original content. In addition to text, images and videos can also be summarised.

NLP (Natural Language Processing) Module is an application to use in the summarization algorithm, and here is the method of how to implement it with Spacy.

These are the following ideas that are helpful to create a summarization algorithm:

Text Preprocessing (remove stopwords, punctuation).

Frequency table of words/Word Frequency Distribution — how many times each word appears in the document

Score each sentence depending on the words it contains and the frequency table

Build summary by joining every sentence above a certain score limit

Let’s start,

1) Import spacy package.

2) Import Text processing package, (STOP_WORDS, punctuation)

3) get the list of the stop words from the spacy.

4) Store the information of the original text in a variable.

5) load the English module of spacy to work on English language data. and create an NLP object.

6) Splitting each word of sentences of the paragraph to create tokens.

7) Iterate through the list and check if the corresponding word is not present in the stop words list and if not, increase its frequency by 1. and create the list variable according to the frequency of the word.

8) At this point having generated the frequencies of each word in the given data, normalize the frequencies by extracting the maximum of all frequencies which was generated in the earlier step and divide frequency of each word by the maximum frequency.

9) Now there is a need to generate scores of each sentence in the data to generate an optimal summary of the given data. For that first tokenize each sentence in the data.

10) Now, having generated a list of sentences and also a list of

words. So, generate a score for each sentence by adding the

weighted frequencies of the word that occurs in a particular

sentence. As it is not interested in a summary, it has to score

only those sentences with less than 30 words.

11) In the earlier step, the output is generated as a dictionary of sentences with their scores. Now, select the top N sentences by passing our sentence scores dictionary along with its values and the required N sentences to the n-largest function in the NLTK library and it turns into a list with sentences according to their sentence scores.

12) Further, convert the sentences to strings and then Finally Join them to generate the finalized summary.

13) Final Output
BERT para responder preguntas sobre SQuAD 2.0 en Español.
By: Jaime Sendra
Tags: Naturallanguageprocessing, Squad, NLP, Bert, Transformers

GitHub | Jaime Sendra

Arctic Code Vault Contributor Dismiss Sign up for your own profile on GitHub, the best place to host code, manage…
TextRank — Unsupervised Algorithm for Keyword Extraction and Text Summarization
By: Prakhar Mishra
Tags: Text Summarization, Naturallanguageprocessing, Research, Gensim, Pagerank

TextRank is a graph-based ranking algorithm under the hood for ranking chunks of text segments in order of their importance in the text document. This research was done in the University of Texas by Rada Mihalcea and Paul Tarau and proved that the results on unsupervised keyword extraction and unsupervised extractive summarization were equally good and better in some cases when compared with previous benchmarks.

TextRank revolves around the idea of representing the concerned lexical unit in the form of graph and later use the famous PageRank algorithm to rank those chunks. For those, who are not sure what PageRank is, read the next paragraph, others can skip to the other one following that.

PageRank is a ranking algorithm that is used to score webpages and index them accordingly. It is named after one of the Google’s founders Larry Page. The equation goes as -

PR(A) = (1-d) + d (PR(T1)/C(T1) + … + PR(Tn)/C(Tn))

Consider A to be a page and T1, T2, ,…. Tn are all pages that are pointing to A.

PR(T1) = PageRank of page T1

C(T1) = Outgoing links from page T1

d = Damping factor (takes care of cutting down too much influence of certain page which might result using fake bots)

1-d = This factor takes care of any new page that has no link pointing to it. Such pages are still to get some PR.

d = 0.85 (As per original paper on PageRank)

Note that PageRank forms a probability distribution over web pages, so the sum of all web pages’ PageRanks will be one.



Authors talk majorly about solving two problems -

Keyword Extraction Summarization

Watch Paper Walkthrough for in-depth understanding — https://www.youtube.com/watch?v=2l6Fa767kEw
tokenization using indic NLP library
By: Raghvendra Pratap Singh
Tags: Naturallanguageprocessing, Hindi, Indic Nlp Library, Tokenization, NLP

Hello! I should say नमस्ते since today’s topic is regarding Indian language.

Natural Language Processing looks fascinating but it’s similar to Machine Learning where we need data cleaning and data pre-processing.

Sounds boring right? 😬 But it’s not our mistake…machines never tried to learn human languages 😐. It was us who generously learnt numbers to communicate with them 🙅. Jokes apart, when we talk data pre-processing, Tokenization is an integral part of this. Basically, we split the text further into units called tokens which can be words or characters.

Fine! we understood this. English has NLTK library but what about Hindi?

Well, thanks to Anoop Kunchukuttan for gifting us with Indic NLP library. For Jupyter notebooks or Google colab, you can install it from here. He has provided a fine notebook as a tutorial where output of tokenization is

Tokens:

सुनो

,

कुछ

आवाज़

आ

रही

है

।

फोन

? Reference -- https://nbviewer.jupyter.org/url/anoopkunchukuttan.github.io/indic_nlp_library/doc/indic_nlp_examples.ipynb 🙏

But this output comes as a list. When we try use this tokenized list in further steps like TF-IDF, we may face an error like:

AttributeError: 'list' object has no attribute 'lower'`

To understand more, check following links on stackoverflow:

What to do now?

Well, I’ve following snippet for you:

from indicnlp.tokenize import indic_tokenize listFinal = [] for i in trainList:

value = indic_tokenize.trivial_tokenize(i)

stringVal = ' '.join(value).lower()

listFinal.append(stringVal)

It worked for me with impressive results. 💯

I have shared my own experience in this article. Please share your thoughts if you find anything incorrect here. 🙏

Twitter: @MrTomarOfficial

LinkedIn: https://ie.linkedin.com/in/raghvendra-pratap-singh-tomar
Clarifai New Product Release | Release 6.10 Features New Model Operators
By: Thomas Molfetto
Tags: Data Science, Artificial Intelligence, Naturallanguageprocessing, Computer Vision

By Jeff Toffoli

Clarifai Release 6.10

We’re excited to announce version 6.10 of the Clarifai platform. We are releasing a new Swift client so that you can now easily integrate Clarifai services on iOS devices, the Clarifai API Collection has been released publicly on Postman, plus we are introducing a ton of new operators that make your workflows more powerful than ever before.

Postman Public API Collection

It has never been easier to work with the Clarifai API. With the Clarifai API Collection you can instantly access hundreds of pre-configured API calls to the Clarifai platform. Now you can easily find what you are looking for, and experiment with and test your ideas right in Postman. Try it yourself.

Swift gRPC Client Beta

Swift is the general-purpose, multi-paradigm, compiled programming language developed by Apple and the open-source community. Now with the new Swift gRPC client you can easily integrate Clarifai services into applications on Apple devices. Find out more.

The Tesseract Operator Beta

Tesseract is an OCR engine with support for unicode. Tesseract provides a new neural net (LSTM) based OCR engine which bring next generation line recognition to OCR text understanding. Try it out in Model Model in Portal.

Isolation Operator Beta

Add the isolation operator to your workflow that contains people detections so that you can understand how social distancing practices are being followed. Try it out in Model Model in Portal.

Barcodes Operator Beta

Turn any camera into a barcode reader. The barcode operator can detect and recognize barcodes from an image. It assigns regions with barcode text for each detected barcode. Try it out in Model Model in Portal.

Learn more about our releases in the Changelog.

We’re excited to announce version 6.10 of the Clarifai platform. New clients, API collections and more.