fabiochiu/medium-articles huggingface dataset:
Getting data from BigQuery to Facebook Ads in under 5 minutes
Tags: Bigquery, Facebook Ads

Getting data from BigQuery to Facebook Ads in under 5 minutes Chris Apr 26·2 min read

You’ve put a lot of work into your customer data warehouse in BigQuery. Your data team has been stitching together source data on your customers for months: mobile app interactions, email opens, transaction history, loyalty program data.

The next big question is how do you generate value from all of this customer data? You can use it in data visualizations in Tableau, Looker, PowerBI of course, but how do you go beyond visualizations?

We’re going to show you how in 5 minutes you can activate your customer data in BigQuery to Facebook Ads to target ad campaigns.

But, first, let’s take a quick step back. Why send your data to Facebook Ads in the first place from BigQuery? We see several common use cases across clients where bringing the power of BigQuery to Facebook Ads can create real business value. Some of the common use cases we see include:

Best Customer Lookalikes — Exporting lists of your best customers to Facebook to find “lookalikes” to acquire. Churn Re-activation — Exporting lists of customers who have churned and you are trying to win back based on what they purchased in the past. Free to Paid Conversion — Exporting lists of free sign-ups you are trying to convert to paid customers. Upsell Non-Email Openers — The majority of your customers don’t read your emails, unfortunately. For those non-readers, you can get in front of them on Facebook for cross-selling and upsell to new products. This is a clever use case we see all the time.

When activating your BigQuery customer data in Facebook there are a few key elements to consider:

One time vs Ongoing Sync — Do you want the customer segment in BigQuery to be sent once to Facebook? Or do you want it to update every day based on which customers now fit your segment criteria (ie. churned customers). Usually, we see customers want ongoing sync, but find it difficult to implement in practice. Measurement — Are you going to hold back any portion of your customer segment as a control group to measure if Facebook is driving incremental revenue?

With Flywheel you can start syncing your BigQuery data to Facebook Ads in a few simple steps:

Send data from BigQuery to Facebook Ads

Setup BigQuery as a Source Create customer segment based on the data you have in BigQuery Select a treatment/control group Export to Facebook Ads

You are done. Every day we’ll load new users that fit your customer segment and remove the ones that no longer qualify. And you have a control group held out for measurement that is stored in BigQuery for you.

If you are interested in how to sync your BigQuery data to Facebook Ads reach out to us at solutions@flywheelsoftware.com.
Optimizing Google BigQuery with Data Studio reports
By: Rohit Singh
Tags: Cost Optimization, Google Data Studio, Firstory, Bigquery, Data Engineering

A Case Study: how to optimize the query

We started monitoring the tables and observing those tables, as we pronounced them as Dimension tables getting bigger which made queries expensive and slow. In our system, GDS reports require data collected daily which needs to be processed each time GB’s data from dimension tables.

We have a table A which is 335.28 GB in size.

big query:

select * from `bigquery-project_id.datasetName.tableA`

— This query will process 335.3 GB when run.

We modified the table A structure and partitioned it based on a date with a clustered column after references this video Data Warehousing With BigQuery: Best Practices:

create table `bigquery-project_id.datasetName.tableB`

partition by partitioned_date -- date columns

clustered by event_name -- columns

options (require_partition_filter=true)

as (

select * from `bigquery-project_id.datasetName.tableA

)

Now run the query to select all columns of last day on both tables for comparison:

Query 1:

select * from `bigquery-project_id.datasetName.tableA`

where date(event_time)>=date_add(current_date(), interval — 1 day)

Query complete (17.7 sec elapsed, 335.3 GB processed)

Query 2:

select * from `bigquery-project_id.datasetName.tableB`

where date(partitioned_date)>=date_add(current_date(), interval — 1 day)

Query complete (21.5 sec elapsed, 959.5 MB processed)

Whaaat! partitioned date column reduces data read 350 times. But wait for a sec, it also increased the read time.

Now run the Query 2 with where clause on event_name.

Query 3:

select * from `bigquery-project_id.datasetName.tableB`

where date(partitioned_date)>=date_add(current_date(), interval — 1 day) and event_name='App Installed'

Query complete (1.9 sec elapsed, 507.3 MB processed)

Can’t Believe It Right! But this is true in 2 seconds, we got the result with the help of cluster column event_name.
“OK Boomer” escalated quickly — a reddit+BigQuery report
By: Felipe Hoffa
Tags: Reddit, Data Visualization, Boomers, Trends, Bigquery

“OK Boomer” escalated quickly — a reddit+BigQuery report

Let’s use BigQuery to find the first time that someone commented “OK Boomer” on reddit. Turns out it happened hours and even years before the alleged first tweet. Was this all an attempt by teenagers to replace the word “old” with “boomer”? Let’s dig in. Felipe Hoffa Nov 9, 2019·5 min read

“OK boomer” growth by subreddit:

# of unique accounts per sub that have commented “OK Boomer”

The first “OK boomer”

According to Know Your Meme this tweet marks the first appearance of “ok boomer”:

The first tweet for “Ok Boomer”. Source: Know Your Meme

And the author has been celebrating it:

But was this the real first appearance? 90 minutes earlier, that same day — April 12, 2018— someone else had immortalized “OK BOOMERS” on reddit:

An “OK boomer” on reddit, 90 minutes before that first tweet.

Turns out there are 9 reddit comments even before that, with “Ok boomer” starting in September 2009:

“OK boomer” reddit prehistory: 9 comments between 2009 and April 12, 2018

And things didn’t escalate quickly back then. It took 9 years to get the first 9 “OK boomer” on reddit, and then we have to jump until October 2018. That’s when we start seeing an almost daily dose of the phrase:

Waiting from April 2018 until October 2018 to find an almost daily dose of “OK Boomer” on reddit.

And it had spread through different subreddits! From /r/Tinder, to /r/worldnews, to /r/xboxone, to /r/golang — redditors started embracing it.

And something happened on March 2019. Until then we had a spotty supply of the phrase, but on March 24 six different accounts decide to comment “OK boomer”. Since that day, the trend continues going up until June 2019, with more than 30 accounts commenting “OK boomer” per day.
GSoC’20 #3 [Week 4]
By: Akshara P
Tags: NLP, Augur, Gsoc, Sentiment Analysis

In the final week of the first phase, I worked on some improvements on the sentiment analysis I had done last week. I analyzed sentiment trends across various repositories and saw that in quite a few, the negative sentiment was dominant!

After discussing with my mentors, we decided on using the Joblib library for storing the trained models, for use in the workers. I have started with the process of converting the message uniqueness and sentiment analysis notebooks into the Message Insights Worker.

Alongside this, as suggested by my mentors, the other task I’m working on now is the qualitative analysis of pull requests — based on the length of the PR, the sentiment of it’s title, the number of commits associated with it, and the sentiment of all the messages pertaining to it. This is being done on short and long PRs separately. I’m looking at gathering other important features. After collecting all the features I will be working with the time-series data to identify anomalous pull requests.

My mentor also suggested another interesting aspect that can be studied — analyzing the flow of information within a repo through conversations! As a first step towards this, I have started collecting a dataset with details of the message contributor and list of personal mentions made in the message for various repositories.

The next phase would comprise of finishing the worker for message insights and deciding how the results can be visualized and shown to the end-users. I will then move ahead with anomalous PRs and issues detection. Following this, I will start interacting with Auggie for gathering feedback and other interesting analysis. More details in the upcoming blogs!

My notebooks can be seen here: https://github.com/chaoss/augur/tree/akshara/messages-notebooks
Analysis of Emotion Data: A Dataset for Emotion Recognition Tasks
By: Parul Pandey
Tags: Machine Learning, NLP, Sentiment Analysis, Exploratory Data Analysis, Transformers

Accessing the datatset

We have seen the dataset. Let’s now see how to access it. I’ll demonstrate two ways to load the dataset and use it.

1. Loading data directly from the source

The dataset is already available as a pandas dataframe. There is an accompanying notebook showing how to use it for fine-tuning a pre-trained language model for emotion classification. It can be easily accessed as follows:

Let’s look at few statistics of data.

data.head()

First few rows of the dataset | Image by Author

We can we the first five rows of the dataset containing tweets and their corresponding labels. From here, we can split the dataset into a test and a validation set and train an emotion classifier. It is also a good idea to do some exploratory text analysis to understand the data. We’ll get to that in a bit, but before that, let me show you another way to load the dataset.

2. Downloading data from the Hugging Face 🤗 Dataset Hub.

Hugging face datasets library provides API to download public datasets and preprocess them easily. You can refer to this video on Huggingface datasets to get started.

The main page of HuggingFace datasets Hub | source: Huggingface website

We’ll first download the datasets library and import the necessary modules

!pip install datasets

from datasets import load_dataset



emotion_dataset = load_dataset("emotion")

This creates an emotion_dataset object.

emotion_dataset

emotion_dataset object | Image by Author

What we get is a dictionary with each key corresponding to a different split. Let’s access the training data to see its contents.

emotion_train = emotion_dataset['train']

print(emotion_train[0])

print(emotion_train.column_names)

print(emotion_train.features)

Training dataset | Image by Author

The training dataset comprises six different classes — sadness, joy, love, anger, fear, and surprise.

Converting emotion datasets into a Pandas DataFrame

We can now easily convert the above dataset into a pandas dataframe and analyze it further.

import pandas as pd emotion_dataset.set_format(type="pandas")

train = emotion_dataset["train"][:]

test = emotion_dataset["test"][:]

val = emotion_dataset["validation"][:]

Let’s quickly check if the conversion was successful.

train.head()

First few rows of the training dataset | Image by Author

Yes, indeed! Now you know how to access datasets from the Huggingface dataset hub- another addition to your data science skill. So what do you do when you have the data? You explore it, and that is precisely what we are going to do in the next section.
Twitter Customer Support Sentiment Analysis and Topic Modeling
By: Kari Davis
Tags: Clustering, Sentiment Analysis, Data Visualization, NLP, Data Science

Sentiment Analysis:

My first step was to examine the overall customer sentiment concerning each of the businesses in my dataset. Sentiment analysis is a part of NLP that tries to extract opinions and attitudes from text. VADER is a sentiment analysis tool that has been pre-trained on social media data to provide an overall sentiment score from -1 (most negative sentiment) to +1 (most positive sentiment). I chose VADER because due to being trained on social media, it would be equipped to handle Twitter text which is very messy and filled with emojis.

I started by grouping all of the tweets in my dataset into full conversations. Then I split out only the customer half of the conversation since the words from the companies were generally the same/helpful in nature and would only fog the overall sentiment towards the business. I fed the text from the customer half of the conversation into VADER and it returns negative, neutral, positive, and compound scores for the text.

I aggregated the compound conversation sentiment scores for each business in my dataset to get an average customer sentiment score for the business. Below is a histogram of the conversation sentiment scores for Walmart, which had the highest positive sentiment score of all the businesses in my dataset at 0.634.

As you can see from the histogram, there were very few conversations with Walmart support that had a negative sentiment overall. I found that most of the tweets involving Walmart were praising their low prices and great deals. There was not much talk in the Walmart tweets about customer service or customers requiring help/attention.

KFC on the other hand, had the most negative overall sentiment in their twitter conversations at -0.251. As you can see in the histogram below, there was more of a spread in sentiment for KFC, with a strong skew towards the negative. Most of the KFC support conversations involved customers complaining about long wait times, bad service, or poor food quality.
A taste of the power of BERT in 3 easy steps
By: Mark Ryan
Tags: Machine Learning, Sentiment Analysis, NLP, Docker, Bert

Dead easy way to try out BERT

One of the exciting aspects of machine learning is how the boundaries are getting moved out all the time. It’s even more exciting if you can get a taste of new technology without having to go through a lot of hassle. Since hearing about the BERT (Bidirectional Encoder Representations from Transformers) language representation model, I’ve wanted to get a chance to exercise it. BERT struck me as an idea that was both elegant and practical. Thanks to this text sentiment classifier you can get a taste of the power of BERT from a standing start in a couple of hours.

After reviewing the text sentiment classifier article, I followed the associated github readme which led me flawlessly through the steps to get the model set up:

Set up Docker — I have a brand new Windows 10 system, so I spent a good portion of the overall end-to-end time getting Docker going. It took a couple of restarts and toggling Hyper-V off and on to get Docker to work. It wasn’t too irritating but it didn’t do anything to dispel my impression of Docker being a diva on Windows. Deploy the model: run this command (e.g. in Powershell in Windows) to run the docker image and start the model serving API:

$ docker run -it -p 5000:5000 codait/max-text-sentiment-classifier

3. Load the model in a browser:

http://localhost:5000

That’s it! Now you’re ready to score some text. To do so, you simply have to enter the text that you want to score in the payload field, with each sample surrounded by double quotes and separated by commas, and click on Execute:

The results show up in the response body field:

You can see for these two randomly chosen examples (from the Amazon video review dataset available here) that the model detects the sentiment accurately.

Other than having to fiddle a bit to get Docker going, the only other hitch I encountered was the need to finesse the sample text a bit in Excel to remove embedded double quotes and line feeds.

If you want to exercise this text sentiment classifier on larger set of samples, I’ve prepared a subset of the Amazon video reviews that you can score with the model. The CSV file contains sample text that is ready to go, along with the positive and negative scores the model produced for each.

Thanks to the text sentiment classifier described in this article, I was able to get a taste of the power of BERT from a standing start in little over two hours. It’s great to be able to exploit a state-of-the are language model with so little hassle.
What now we know about the past and the future? Part 2
By: Parliamentary Voices
Tags: Big Data, Sentiment Analysis, Digital Humanities, Dhh19, Social Studies

“The fact that the outermost regions exist is tied up to a large degree with colonial history. In the Middle Ages, those regions and their people did not belong to Europe, and even today, they are not really considered to be European.” (Erik Meijer, 25 October 2000, Strasbourg)

In the previous post we discussed our findings from the concept network constructing and topic modeling. Now we present the results of sentiment analysis and close reading.

Sentiment analysis

This method identifies the sensitivity and the subjectivity of text using special algorythm. It estimates the sensitivity of the text by the scale from -1 to +1 that helps us to undestand whether the message is positive, negative or neutral. This tools also evaluates the subjectivity of the message from 0 to 1 according to proportion of facts and opinions in the text.

We used the sentiment analysis in order to research how MPs express their emotions while speaking about the past and the future. The plot below illustates all the sentences from the subcorpora. We could see that in the average speeches tend to be neutral but at the same time speeches about the past have more variations in sensitivity and subjectivity according to the speeches about the future.

A possible explanation for this is that the past and events from the past may be a subject of speculation in the political discourse. Depending on the speaker’s intention, the various ethical evaluation may be given to the historical events, experience and persons.

These plots also demonstrates that the variation of the sentiments in the speeches about the past is wider than in the speeches about the future.
Training your own Sentiment Analyzer with spaCy
By: Dipayan Mukhopadhyay
Tags: Sentiment Analysis, Machine Learning, Spacy, Lstm

Hi folks!!! In this blog I am going to discuss about training an LSTM based sentiment analyzer, with the help of spaCy. Recently I was working on twitter sentiment analysis and I have spent quite a long time exploring already available pre trained models for that purpose. I came across python libraries like TextBlob, VaderSentimentAnalyser, Flair etc. Though these libraries work pretty well, we cannot tweak them as per our need. I was looking for something specific to my use case. I was in need of a model which will be trained with my own dataset. At the same time I did not want to be worried about nitty gritty details like word embedding, network architecture etc. spaCy came to the rescue. They have provided an outline to train an lstm model with decent word embedding. All we need to do is passing the data properly with minimal change in code. Let us start.

First we have to fetch the python script provided by spaCy from below github link and store that file.

Prerequisites:

pip install spacy spacy download en_vectors_web_lg

## en_vectors_web_lg is the pre trained Word Vector Model spaCy is providing pip install keras==2.0.9 Compatible with: spaCy v2.0.0+

Data Preparation

If we run deep_learning_keras.py file without feeding it with any data, it will by default download imdb reviews dataset and train the model with that. As we want to train with custom data we need to process the data and maintain a particular file structure.

Suppose our data is in following format. Where 1 is Positive sentiment, 0 is Negative Sentiment.

After performing some basic pre-processing like removal of punctuations, special characters, urls, we need to split the data into two parts(Train Test Split), both named same(for example Tweet_Sentiment.csv), one residing in train folder another in test folder. Also we have to create a folder to store final model weights(Here model_lstm).
Stacking machine learning models for speech sentiment analysis
By: Corentin Garet
Tags: Machine Learning, Deep Learning, Sentiment Analysis, Stacking, Data Science

How to build a model that recognizes human sentiment from audio and text recordings. Co-authors: Qianwen Guan, Alexandre Laurent.

In the context of the final project of Le Wagon’s bootcamp, my team and I decided to take on a fascinating task: Speech sentiment recognition.

It is a great challenge to take up because emotions are subjective on the lines of culture, gender, language and even down to the individual person, it is therefore difficult to universally classify human sentiment.

Our data

We found a data set from Carnegie Melon University called CMU-MOSEI¹ which is the largest dataset of sentence level sentiment analysis and emotion recognition in online videos. It contains more than 65 hours of annotated video from more than 1,000 speakers and 250 topics.

The data was divided between segments of variable lengths, each representing a full spoken sentence (features), and the sentiment, our target, which varied between the values -3 to 3 (from negative to positive, 0 being neutral).

We decided to analyse both audio recordings and text transcripts to predict the sentiment behind a person’ sentence. Our intuition was that combining two models with two different sources, using multi-modal learning, could improve our performance.

Data preprocessing

The first step our work what to clean both text and audio data.

While the text was already extracted from the videos, the editing there essentially consisted in doing basic formatting for the text files (removal of punctuation, numbers and uppercase). However, in Natural Language Processing (NLP), it is difficult to chose what parts of text to remove and what parts are to be kept (single words, sentences, whole conversations). I attempted to lemmatize and stem words but no improvements in performance were found.

Audio format was slightly more complex, we tried two approaches:

1. The first was audio feature extraction using Python’s library librosa. It enables to extract 5 major features (mean values) from each recording: MFCC, Chroma, Mel Spectrogram, Spectral Centroid & Tonnetz. From those, I obtained around 190 features which could then be used for modelling as tabular data.

2. The second was audio to Mel spectrogram which allowed us to interpret audio as an image and to model it from a visual point of view. Looking at the following figure, the x-axis represents time (s), the y-axis frequency (Hz), and the color intensity represents the amplitude of the signal (dB). In this case, the images allowed for feature extraction to occur in a deep learning setting (convolutional network).
Real-Time Tweet Analysis for Covid-19 Vaccine using Python
By: Ayça Erbaşı
Tags: Sentiment Analysis, Covid 19, Twitter, Python, Data Science

Photo by Joshua Hoehne on Unsplash

Real-time tweets about the covid-19 vaccine have been analyzed in this article. My analyzes give answers to whether people think positive or negative about the covid-19 vaccine, or are neutral. You will see which words are tweeted the most with the covid-19 vaccine. Also, tweets about the types of vaccines were compared.

I got the tweets with my Twitter developer account and used Python.

The Data

I started with store authentication credentials in relevant variables. I limited the last 1000 tweets.

I filtered tweets to track only these keywords; ‘covid19 vaccine’, ‘covid-19 vaccine’, ‘coronavirus vaccine’, ‘covid19 vaccines’, ‘covid-19 vaccines’, ‘coronavirus vaccines’, ‘vaccine’, ‘vaccines’.

After printing, the keys of the first tweet dict has been seen. It includes created time, id, location, user, status, in reply to status id, in reply to the user id, retweeted status, coordinates, retweeted count, reply count, favorite count, etc. In this case, the text and language were added as a feature.

Let’s quick glance at the data.

Figure 1: The data frame.

Figure 2

Figure 2 shows an example of a tweet about the vaccine. The first row shows the tweet, it’s been retweeted from the user of dsa_bill. The user id is shown in the second row and, respectively, his / her user name, the number of followers of this user, the location, and the description of the account were shown.

Sentiment Analysis with Vader

Sentiment Analysis falls into the broad category of text classification, the classifier is supposed to tell if the sentiment behind that is positive, negative, or neutral.

Figure 3: An example of positive and negative tweets.

Figure 4: The new data frame.

The feature of Sentiment Score, Subjectivity, Polarity, and Analysis added to the data frame. Here are the number of Positive, Neutral, and Negatives tweets.

Figure 5

It’s more clear to see the difference with a graph.

Figure 6: Sentiment Analysis graph.

The tweets about Pfizer, Moderna, and Coronavac compared. Here is the proportion:

Word Cloud

The most-tweeted words in texts containing the covid-19 vaccine are possible to see with word cloud. To make the word cloud clear, I stopped these words; “covid”, “covid19”, “vaccines”, “vaccine”, “https”, “will”, “coronavirus”, “rt”. Words limited to 20.

Figure 7: The word cloud.

Conclusion

As a result, the analysis was made over the last 1000 tweets about the covid-19 vaccine. There are 518 neutral, 348 positive, and 134 negative tweets. The proportion of tweets mentioning Pfizer is 0.05%, 0.015% for Moderna and 0.003% for Coronavac.

Thank you for reading.

Please don’t hesitate to give me feedback.
India Budget 2019 Sentiment analysis using nltk rule based approach
By: Jithin James
Tags: Budget, Sentiment Analysis, Scraping, Nltk, India

This is a small attempt to do sentimental analysis on latest Indian Union budget presented by Finance Minister Nirmala Sitharaman on July 5th 2019.

Data for sentimental analysis is crawled(using ‘requests’ library) from a Times of India article(https://timesofindia.indiatimes.com/business/india-business/budget-2019-highlights-gst-rate-on-electric-vehicles-to-be-lowered-from-12-to-5/articleshow/70084365.cms). The number of comments taken from this article was 300.

The sentiment analysis was done by Rule based method using nltk’s SentimentIntensityAnalyzer with the vader_lexicon. As the labelled data was not available, Rule based method was adopted instead of a learning based approach to classify the sentiments

Wordcloud on Neutral comments

Neutral comments

Wordcloud on Positive comments

Positive comments

Wordcloud on Negative comments

Negative comments

From all the segments, it seems one of the highlighted words is ‘middle class’ or ‘salaried class’ . One reason might be the viewership of this article is mostly this class and are discussing mainly on the Income tax related points.

And ‘Modi’ seems to be the most highlighted word in the positive sentiment section.

Top engaged locations

Based on the available location data of the engaged users, below graph depicts the top 8 locations/cities of the users and sentiments from the cities.

top 8 city sentiments

Looks like Bengaluru is unhappy with the budget while most other cities seems to be happy. Could be owing to the reason that Bengaluru has a good population high salary earning individuals.

Sentiments on most liked or disliked comments

Below is an analysis of the top 15 comments that were liked/disliked and the sentiments of the same.

top 15 liked comments sentiments

top 15 liked comments sentiments

top 15 disliked comments sentiments

top 15 disliked comments sentiments

Looks like, the top liked/disliked comments are both of the ‘negative’ comments on the budget, however a majority of the top disliked comments seems to be on the negative comments than the positive or neutral.
PREDICTING NIGERIA 2019 ELECTION WITH AI — SENTIMENT ANALYSIS
By: Kelly Idehen
Tags: Sentiment Analysis, Election, Politics, AI, Data Science

By streaming data from Twitter in real-time we were able to gather over 800,000 tweets which formed the sample for this analysis.

Data (Tweets) streamed were primary keywords pertaining to the Nigeria 2019 election such as: “PDP”, “Atiku”, “Atikulated”, “Peter Obi”, “APC”, “Buhari”, “Mbuhari”, “Tinubu”, “Osinbajo”, “Lagos”, “Kaduna”, “Kano”, “Abia”, “Delta”, “port harcourt” .

The sentiments of each Tweet was resolved and all values were stored on a database. See data description below.

Tweets ordered by sentiments.

Overall the percentage of Negative and Positive Tweets are 42% and 58% respectively as can be seen from the chart below.

General Pre-Election Sentiments.

For the Individual presidential candidates, Buhari had a higher number of overall mentions in tweets totaling about 218,519 tweets. Atiku on the other hand had a total mention of 136,149 mentions in tweets.

The distribution of tweets for each presidential candidate is as given below.

As can be seen in the distribution, Atiku has a higher positive rating in percentage when compared to tweets that mentions him. But this is not the conclusive milestone on our data.

Next we explore sentiment distribution as regards to the two main political parties in question, APC and PDP with a total tweet mention of 81,312 and 75,642 respectively.

In the next distribution we will be looking at comparing related data side by side to give us a holistic view of what is really happening.
How’s 2021 going? Twitter Sentiment Analysis (NLP)
By: Sandra Machon
Tags: Sentiment Analysis, Data Visualization, Twitter, Data Science, Vader

Tools and workflow

This diagram shows the project flow:

Project flow diagram

The tools used include Tweepy (for mining tweets), Pandas (for data cleaning/wrangling), Tweet Preprocessor (for rapid tweet cleaning), NLTK (for tokenization, stopwords removal and POS tagging), Plotly, Matplotlib and Word Cloud (for visualization). You can check the remaining built-in libraries and detailed explanations in my source code on GitHub.

Tweets Mining

I used a Python library Tweepy to build a tweets dataset from scratch. Tweepy works with Twitter API and in order to use it, you need to start a Twitter developer account. That’s how you can get a unique consumer key/access code that you need to access the Twitter data. Tweepy is a pretty straightforward library to use, however scraping tweets can be slow due to the new mining limits which Twitter imposed through their API. That’s why Tweepy introduced the “wait_on_rate_limit” method, which “puts the scraper to sleep” when the Tweet’s limit has been reached. I scraped 37,621 tweets in about four hours.

This is how the authorisation code looks like:

I created the following function to scrape the tweets:

With this function, you can easily configure a searched phrase, number of tweets and time frame. You can also set up an optional argument to start scraping from the latest tweet’s ID and avoid duplicates.

The function returns a .csv file with “tweet text”, “tweet ID”, “Time Created”, “Location”, “Retweets” and “Number of Likes”. To see other embedded Twitter parameters run:

This script scrapes one tweet and then prints its parameters.

Data Cleaning/Exploration

In this case, data cleaning was a short step. Firstly, I checked the shape of the data and the name of the columns. I dropped a column called ‘Unnamed: 0” which contained a second index (created during merging). Secondly I checked for any duplicates with the column “id” as a subset and dropped all of them (23 in total). Lastly, I checked for NaN values and since ‘Location’ had 37,643 missing values I decided to drop that column. After a bit of research, I found out that Twitter disabled automatic location tagging from their users. That’s why the location data is missing for most of the tweets.

This is how the output data frame looks like:
Sentiment Analysis on The New York Time' Coverage on City Lockdown In Italy And China
By: Ann Cheung
Tags: Sentiment Analysis, New York Times, Italy, China, Coronavirus

Sentiment Analysis on The New York Time' Coverage on City Lockdown In Italy And China Ann Cheung Mar 18, 2020·2 min read

The New York Times’ tweets on March 8 about the locking down cities of Italy and China caused controversy in China.

The tweets were sent on the same day, 20 minutes apart, and appeared in tandem on the same page. Both tweets describe the measures taken by the country to block cities in response to the coronavirus. The New York Times described China’s actions as a sacrifice to people’s livelihoods and liberties while saying that Italy was making efforts to contain the outbreak.

Such tweets started a lot of conversation and emotion on the Chinese Internet. Online comments, including the People’s Daily, called the New York Times a double standard media.

Since the outbreak, the New York Times has published five articles on China and five on Italy regarding the city blocking.

Sentitives terms proportion in reports about China

In the New York Times’ reports on China, negative words consistently account for more than positive words, ranging from 66.1% to as high as 84.6%. The coverage was mainly about people’s anxiety and anger, government mistakes, questions about the effectiveness of quarantines, and so on.

Sentitives terms proportion in reports about Italy

On the other hand, in the New York Times’s coverage of Italy, the proportion of negative words was generally higher than that of positive words, but the overall difference was not significant. The percentage of negative words was about 60%, and the highest was 72.2%. There was also criticism of the government and relatively less mention of the opinions of the citizens except for a report about the “balcony concert.”

When we put the sensitive terms proportion of both countries together, it is obvious that negative words percentage are generally higher in reports about China other than Italy.

As on twitter, one of the main points about covering China is the violation of humanitarian. In its reports of the lockdown, it cited “a public health experiment,” “easily backfire,” and “a disaster. “ The same measure in Italy was described as “ the most sweeping effort outside China to stop the spread of the coronavirus, “ and “save it from the ravages of the virus in the long term. ” At the same time, The New York Times called it militaristic when the People’s Daily use “people’s war” to describe the fight against the epidemic, but the word “war” has also been used in its reports about Italy, one of them is directly quoted from a doctor.

Whether the New York Times was intentionally selective in its coverage is not clear from the current data. But it’s safe to say that there are different versions of similar situations in its coverages.
Sentiment analysis by using Twitter data
By: Sachin Katageri
Tags: Machine Learning, Sentiment Analysis, Twitter, Data Science, Artificial Intelligence

Sentiment analysis (a.k.a opinion mining) is the automated process of identifying and extracting the subjective information that underlies a text. This can be either an opinion, a judgment, or a feeling about a particular topic or subject. The most common type of sentiment analysis is called ‘polarity detection’ and involves classifying a statement as ‘positive’, ‘negative’, or ‘neutral’.

For example, let’s take this sentence: “I don’t find the app useful: it’s really slow and constantly crashing”. A sentiment analysis model would automatically tag this as Negative.

steps:

Gather Twitter Data Prepare Your Data Create A Sentiment Analysis Model Visualize Your Results

In this section, we’ll explain each of these stages and provide tools for both coders and non-coders so you can get started with sentiment analysis right away.

1. Gathering Twitter Data

It’s important that your data is representative of what you’re trying to find out because you’ll use it to:

Train your sentiment analysis model

Test how your model performs on Twitter data

Dataset link :

2. Prepare Your Data

Code link :

model links:

https://drive.google.com/file/d/1WMen1r0M-K45gN1-QmBQh2-C6NSt9jbZ/view?usp=sharing

3. Create a Twitter Sentiment Analysis Model
Multi-product service desk and employee experiences?
By: Iswarya Rezolve
Tags: Ai Service Desk, Ai Services, Advanced Ai Service Desk, Service Design

A multi-brand service desk provides an integrated experience for multiple departments or support channels while maintaining individual identity. Let’s dive deeper into the multi-brand service desk and enhance the employee experience in the new remote work culture.

What is a multi-brand Service Desk?

In general, multi-brand support portals can be managed and managed centrally within a single helpdesk implementation. This means you can offer different content and brands for each support portal.

It is a great advantage for many organizations looking for a help desk that scales to use the same support tool and staff across multiple portals and maintain full control over workflows, branding, and content.

What is the use of a multi-brand helpdesk?

Using a multi-brand helpdesk can provide great value in many situations and situations:

Internal and external support

It helps internal employees when they have a question or need support and helps new customers who come to the page.

Numerous services

Provide differentiated support across departments and categories for various inquiries and requests.

Multi-brand

It serves as a self-service portal that allows people to get the information they need in 15 seconds, unlike traditional and outdated SharePoint content, which means that tickets are automatically generated if they feel their information is unsatisfactory.

A / B Test

Experiment to find out which portal supports, content, and processes get the most engagement with employees.
Beta luck next time
By: Alice Kershaw
Tags: Grantmaking, Service Design

A year on from March 2020, there are endless articles about what we were all doing a year ago and how we have all fundamentally shifted and changed since then. I am going to write about a small piece of my world which changed, our investment service beta.

This time last year, as we all locked down and started working from home, the first ever service designed, user focused glimpse of the future of investment services at the Fund was open to applications. The beta service we opened in Scotland covered Lottery grants of £3,000 — £10,000, and instead of going through our legacy system, went through the new Investment Management Service or IMS. Launched in January, the intention was that it was going to show us if all our planning, research, designing, listening and building had created the right service for real people.

Around eight weeks passed, we had even got six new applications in! We knew many tens more had registered and were likely to put in applications. But the world it was launched into was about to shift fundamentally, and our approach with it.

The service we had launched was for Lottery grants following the process set out in the Strategic Funding Framework of the Fund. When the Fund stopped the ‘open programmes’ to focus on supporting the sector, the beta was shut with all of the rest of the rolling grant schemes. We focused the effort of the whole team onto emergency funding initiatives up until the autumn.

Since autumn we have restarted, this has included revisiting our assumptions and looking afresh at the technical design and delivery. We took the learning from the emergency grants processes and the new approach from the Fund. Alongside this we revisited our timetables for delivery and the linked budgets and benefits. Not everything stayed, in some cases we have thrown things out and started again. We have not been able to do training face to face for staff so we have had to bring in extra support.

And now we are back almost to where we were a year ago. Once again the beta is open, this time in Northern Ireland as well as Scotland to help us get more applications into the real thing. Things have shifted and changed for users, for the team delivering it, and across the world since the first one. It is not the same beta re opened, it is a different beta that has learnt from what we launched before and responded to what has happened since. I am excited to see how this one fares.
The Service Designer’s Road to Hell
By: Sean Duffy
Tags: Human Centered Design, Design Thinking, Ethics, Service Design, Product Design

Image Credit: Andrew Herington

If you are in the design world and haven’t read Milton Glaser’s Designer’s Road to Hell, you really should. As poignant as the title is, some of the items are pretty striking in their innocuousness—what does it mean to design a package that looks larger on shelf ? Isn’t that the point of package design, to grab attention, to attract and outdo its neighbors, and ultimately to be bought more than others?

Beyond the examples that might get lost outside the practice of package design, Milton means to illustrate a continuum of ethics around communicating something as you want people to perceive it, versus what the product is actually capable of contributing. Aesthetics and product design and user experience are all in their own rights forms of communication, and all forms of communication are equally able to be used to inspire, clarify, confuse or mislead.

In my read of Milton’s list I have admit that I love the drama inherent in the later, more hellish steps. In #11— Designing a brochure for an SUV that flips over frequently in emergency conditions and is known to have killed 150 people—he connects the benign act of designing a brochure to the fact that even a brochure, well designed, can lead people to a harmful product. I’d summarize Milton’s list with the maxim:

Know thy product, and to thy product be true.

I don’t look at his illustrative list and think “Whew—thank goodness I haven’t designed any 9/11 buttons minted from ground zero wreckage.” But I have reflected on some products I’ve designed for where I wasn’t quite sure if the benefit I was instructed to communicate was or even could be proven. Should the adverts I did for a well-known, predatory financial company have been designed to the best of my abilities? When I try and draw parallels to my own efforts his list seems much less dramatic and frankly personal. Which I believe is what Milton really wanted with the list: not an arbitrary ethical scale to judge fellow designers against but a prompt to consider what outcomes we’re directing our audiences to and through that be a better investigator of the products we work on.

On service design

When it comes to considering the outcomes we direct our audiences to—Service Design requires a more complex understanding of the paths users might end up. It’s beyond approaching a package on a shelf—it’s the shelf itself, what interactions the customer might have as they attempt to purchase, and even what neighborhoods that store was built in.

In Milton’s example it’s pretty easy to connect an unethical product to action that shouldn’t be taken. However, even if the outcome of a service is inherently a public benefit, can the service design be unethical? Definitely. Can we find examples as illustrative as Miltons? Hell yeah (looking at you for-profit healthcare systems).

To that I propose the Service Designer’s Road to Hell, to complement Milton’s list.
Sustainable Food Packaging in Bengaluru
By: Abhipsa Sadangi
Tags: Communication Design, Awareness, Ecodesign, Experience Design, Service Design

UNDERSTAND

Problem Statement

“Increase in the usage and wastage of plastic food packages after the rise of food delivery startups in India.”

Context of the Problem

Food delivery mobile applications and websites have been one of the major causes in the rise of plastic food packaging, specially in the first tier and second tier cities in India. Since we are based out of Bengaluru, we decided to study the scenario in this city, Bengaluru.

Secondary Research

On March 2016, the Karnataka government had prohibited the manufacture and also the sale of thermocol and micro-beads. However, the ban has been circumvented as plastic is well factory-made and accessible within the market. The Karnataka government had written a notification to ban single-use plastics within the state in October 2015 and also the cabinet approved the notification in March 2016 and the ban came into result shortly once.

The Bruhat Bengaluru Mahanagara Palike (BBMP) is currently taking initiatives to enforce the ban in a very stricter manner. It’s currently fastened penalties on all those found producing, supplying, storing, transporting, selling, distributing plastic.

According to the civic authority, plastics are accountable in contributing 20% of the 4,000 tonnes of municipal solid waste generated monthly in Bengaluru. (Figures according to July, 2019)

The items lined within the plastic ban are plastic covers, cups, spoons, plates, flags, banners, buntings, flex, cling films, things manufactured from thermocol (polystyrene).

Primary Research

Primary Research based on our insights of activities happening on M.G Road, Bengaluru

Actionable Insights

Building of the actionable insights

Research study and derivation of Actionable Insights

After completing the Primary Research we went onto doing a detailed analysis of the information gathered. This was done through the pattern of PACT Framework i.e. People, Activity, Context, Technology and segregated our insights and findings accordingly.
The McDonalds brothers low-fidelity prototyping their Speedee Service System
By: Abhay Rautela
Tags: Prototyping, User Experience, UX Design, UX, Service Design

Prototyping is an amazing method to quickly iterate a design from a rough to efficient stable state, which can then be taken and materialised as a product or service. It allows you to get a feel of what the product or service will be like, enough to iron out major issues, which you can then take on to start building for real. You start prototyping right from day 1 of the idea. Do it on paper, do it on a whiteboard. We humans are visual creatures. What we can see, and feel, we understand better than just in form or spoken or written words.

Here is an example I often share to build an understanding of the value of prototyping during UX primer workshops for other company roles, clients or stakeholders because it’s just such a fun thing to watch, and thus does a good job of driving home the benefit of prototyping.

The video shows The McDonalds brothers low-fidelity prototyping their interface (kitchen) for a service (McDonalds- burgers and fries!). It shows how a quick simple (virtually zero-cost to execute) simulation helps uncover service efficiency gaps which they then use to make improvements, to what would ultimately be their revolutionary Speedee service system for fast food production.
Put a different price per country
By: Daniele Catalanotto
Tags: Customer Service, Customer Experience, Pricing, SaaS, Service Design

As I’m running a small business that sells course internationally I learned something very simple but that I never thought about much: one price worldwide doesn’t make any sense.

Pricing is a service design problem

I received a chat message on my website. The person was from Iran. She asked me if it was possible to have a discount for a course and explained that for her country the price of the course was just too big. I live in Switzerland one of the richest countries on earth.

So when I set the pricing for my courses I create a pricing that makes sense to pay my bills. But in other parts of the world what is a reasonable price here is something impossible to pay.

A few solutions

Companies like PoiTnt Free found a simple hack to this problem: regional pricing. For certain countries they offer up to 50% discount.

For my own little business I tell people to contact me in order to adapt the pricing to what makes sense to their context. For people who lost their job and are nearly broke they can get the courses for free. For people in certain countries where paying in US dollars is difficult I offer discounts.

Why regional or flexible pricing is smart service design

You might think: but this is hard to implement! A regional pricing has to be adapted to each currency. Sure you could do it like that. Or you could just ask people to mention how much they can pay based on their situation.

This is a smart service design idea because these people would never have paid the full price for sour service or product. With flexible pricing you make your services and products available to a bigger audience, get some extra cash and you understand that you might leave in a country with a vers favorable situation.

An example

If you want a good example of how to handle pricing per country check out how the guys at Point free did it.