Using MALLET LDA to Learn Why Players Hate Pokémon Sword /Shield | by Adeline Ong | Towards Data Science

The latest Pokémon game for Nintendo Switch launched with fanfare, promising to wow fans with stunning visuals and an open-world adventure. Despite praise from critics, the game was a let-down for most players. In this article, I will walk you through how I used MALLET Latent Dirichlet Allocation (LDA) to find out key reasons why players didn’t like the game.
I used BeautifulSoup to scrap Pokémon Sword and Shield user reviews from Metacritic (I’ve detailed the steps here). Metacritic reviews have rating scores that indicate users’ overall evaluation of the game. This makes our lives easier since we won’t need to run a separate sentiment analysis to find negative reviews.
In total, I collected 3,261 user reviews that were posted from 15 to 28 November 2019. The dataset contained these features:
name: reviewer's username
date: date of review
rating: numeric score ranging from 0 to 10 that gives an overall evaluation of the game. Higher scores indicate a more positive experience.
review: actual review text
The following packages were used for data cleaning and analysis: numpy, pandas, regex, string, nltk, langdetect, sklearn, spaCy, gensim, pprint (optional), matplotlib, and collections.
You will also need to install gensim’s wrapper for MALLET LDA, if you don’t already have it.
Importing logging is also optional, but it’s a best practice as it enables you to have a sense of what’s happening behind the scenes when the model is running.
import numpy as np import pandas as pdimport re import stringimport nltkfrom langdetect import detectfrom sklearn.feature_extraction.text import CountVectorizer import spacyfrom spacy.lang.en.stop_words import STOP_WORDSimport gensimfrom gensim import corpora, models, matutilsfrom gensim.models import CoherenceModelimport osfrom gensim.models.wrappers import LdaMalletmallet_path = '/Users/adelweiss/mallet-2.0.8/bin/mallet'from pprint import pprint #optionalimport matplotlib.pyplot as plt%matplotlib inlinefrom collections import Counterimport logging #optionallogging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
Label the reviews positive, negative or mixed. Metacritic considers scores less than 5 as negative, those 5 to 7 as mixed, and anything above 7 as positive. In the code below, I defined a function to label the reviews based on these rules.
Label the reviews positive, negative or mixed. Metacritic considers scores less than 5 as negative, those 5 to 7 as mixed, and anything above 7 as positive. In the code below, I defined a function to label the reviews based on these rules.
def sentiment(x):    if x > 7:        return 'positive'    if x < 5:        return 'negative'    else: return 'mixed'df['sentiment'] = df['rating'].apply(lambda x:sentiment(x))
Drop duplicate user reviews. Some users posted the same review for both Pokémon Sword and Shield (which is expected, since both games only slightly differ in their content).
Drop duplicate user reviews. Some users posted the same review for both Pokémon Sword and Shield (which is expected, since both games only slightly differ in their content).
df.drop_duplicates(subset='name', keep = 'first', inplace = True)
2. Filter out non-English reviews. To do this, I used langdetect package, which relies on Google’s language detection library and supports 55 languages. The detect function reads a text input, calculates the probability of the text being in the supported languages, and returns the language that has the highest probability.
def language_detection(x):  result = detect(x) if result == 'en':return x  else: return np.NaN  df['review'] = df['review'].apply(lambda x:language_detection(x))
3. Remove stopwords. I combined stopwords from nltk and spaCy, and added custom stopwords such as ‘game’, ‘pokemon’ and ‘pokémon’.
The custom stopwords were added after inspecting each word’s review frequency (e.g. a frequency of 5 means that the word appeared in 5 reviews. More about this in Step 6.). The custom stopwords appeared in a large majority of reviews. Their high frequency across reviews implies that that they are not useful for topic modeling. Consequently, I removed them in subsequent runs to improve model performance.
The code creates a list of stopwords. These words will be removed from the reviews in the next step.
nltk_stop_words = nltk.corpus.stopwords.words('english')stop_words = list(STOP_WORDS)stop_words.extend(['game','pokemon','pokémon']) ### these are common words that appear in almost all reviewsfor word in nltk_stop_words: if word in stop_words: continue else: stop_words.append(word)
4. Text cleaning. This is a list of standard text cleaning steps mostly using regex:
Cast all words in lowercase
Remove words from a list (we will use this to remove stopwords, and rare words that appear in less than 4 reviews)
Remove digits
Remove punctuation
Replace words (to standardize inconsistencies in spelling that I noticed such as ‘game freak’ and ‘gamefreak’)
Remove ‘\r’ string literals
Strip extra whitespaces from words (i.e. two whitespaces in a row), which result from the functions that remove digits, punctuation and string literals
def make_lower(text): return text.lower()def remove_words(text,wordlist): for word in wordlist: if word in text.split():     text = re.sub(r'\b{}\b'.format(word), '', text)  return textdef remove_digits(text): return re.sub('\d', ' ', text)def remove_punctuation(text): text = re.sub('[%s]' % re.escape(string.punctuation), ' ', text)  return re.sub(r'[^\w\s]', ' ', text)def strip_extraspace(text): return re.sub('\s\s+',' ', text)def replace_word(text,word,replacement): return text.replace(word,replacement)def remove_r(text): return text.replace('\r',' ')#df['review'] = df['review'].apply(lambda x:remove_punctuation(x))def clean_text(text): text = make_lower(text) text = remove_punctuation(text) text = remove_digits(text) text = replace_word(text,'game freak','gamefreak')  text = replace_word(text, 'game play', 'gameplay') text = remove_words(text,stop_words) text = remove_r(text) text = strip_extraspace(text) return textdf['review'] = df['review'].apply(lambda x:clean_text(x))
5. Lemmatizing. I used spaCy’s lemmatizer with parts of speech tagging to reduce the reviews to nouns and verbs. Using only nouns and verbs makes it easier to discern what the topics are about.
The code below loops through each word in a review and returns words that are tagged as ‘nouns ’or ‘verbs’. It skips through words that have been lemmatized to ‘-PRON-’ which is what spaCy returns when a word is a pronoun, as well as lemmatized words that are in the stopwords list.
sp = spacy.load('en_core_web_sm')def lemmatize_words(text, allowed_postags=['NOUN', 'ADJ', 'VERB', ‘ADV’]): text = sp(text) lemmed_string ='' for word in text:     if word.pos_ in allowed_postags:         if word.lemma_ == '-PRON-' or word.lemma_ in stop_words:  ### skip words that are not in allowed postags or becomes a    stopword when lemmatised              continue          else: lemmed_string = lemmed_string+' '+word.lemma_     return lemmed_string.lstrip()df['review'] = df['review'].apply(lambda x:lemmatize_words(x, allowed_postags=['NOUN', 'VERB']))
6. Removing rare words. Rare words that do not appear in a substantial number of reviews are also not useful for topic modeling. As with stopwords, I removed them to improve model performance.
This code counts the number of times a word appears across reviews, and appends words that appeared in less than 4 reviews into a list.
word_frequency = Counter()for text in df.review: text = text.split() word_frequency.update(set(text))rare_words = []for key, value in word_frequency.items(): if value < 4: rare_words.append(key)df['review'] = df['review'].apply(lambda x:remove_words(x,rare_words))
7. Select only negative reviews, and create tokens. Finally, since we’re only interested in negative reviews, I filtered the reviews based on their sentiment labels and applied sklearn’s CountVectorizer to create word tokens.
CountVectorizer’s token_pattern argument specifies that only words that have at least 3 characters should be included. My assumption is that 1 and 2-letter words are not very informative, and less useful for topic modeling. Also, the ngram_range argument is set to (1,2) to include bigrams as tokens.
negative = df[df['sentiment']=='negative']vectorizer = CountVectorizer(stop_words=stop_words, ngram_range = (1,2), token_pattern="\\b[a-z][a-z][a-z]+\\b") 
To use MALLET LDA, we’ll need to fit and transform the data using the vectorizer and create some variables that the model needs. First, we fit the CountVectorizer to our negative reviews. Then, we create a document-word matrix and convert it from a sparse matrix into a gensim word corpus. Following this, we create word2id and id2word variables that match words to their numeric token ids and vice versa and save these variables into a dictionary.
vectorizer.fit(negative.review)doc_word = vectorizer.transform(negative.review).transpose()corpus = matutils.Sparse2Corpus(doc_word)word2id = dict((v, k) for v, k in vectorizer.vocabulary_.items())id2word = dict((v, k) for k, v in vectorizer.vocabulary_.items())dictionary = corpora.Dictionary()dictionary.id2token = id2worddictionary.token2id = word2id
To choose the number of topics, we’ll calculate coherence scores for each number of topics specified.
Coherence scores assess the quality of the topics by examining the degree of semantic similarity between each topic’s top words. The higher the score, the better the model.
To calculate the coherence score for each mode, I used the code below that I found here under Section 17. The code loops through a range of numbers, representing the number of topics to apply to the model, calculates the coherence score for each model, saves the scores and plots them.
def compute_coherence_values(dictionary, corpus, texts, limit, start=2, step=3):    """    Compute c_v coherence for various number of topics    Parameters:    ----------    dictionary : Gensim dictionary    corpus : Gensim corpus    texts : List of input texts    limit : Max num of topics    Returns:    -------    model_list : List of LDA topic models    coherence_values : Coherence values corresponding to the LDA model with respective number of topics    """    coherence_values = []    model_list = []    for num_topics in range(start, limit, step):        model = gensim.models.wrappers.LdaMallet(mallet_path, corpus=corpus, num_topics=num_topics, id2word=id2word)        model_list.append(model)        coherencemodel = CoherenceModel(model=model, texts=texts, dictionary=dictionary, coherence='c_v')        coherence_values.append(coherencemodel.get_coherence())    return model_list, coherence_values# Can take a long time to run.model_list, coherence_values = compute_coherence_values(dictionary=id2word, corpus=corpus, texts=data_lemmatized, start=2, limit=40, step=6)# Show graphlimit=40; start=2; step=6;x = range(start, limit, step)plt.plot(x, coherence_values)plt.xlabel("Num Topics")plt.ylabel("Coherence score")plt.legend(("coherence_values"), loc='best')plt.show()
According to the graph, 2 or 5 would be good numbers to choose from since their coherence scores are the highest. However, choosing 2 topics might be oversimplifying things, so let’s go with 5.
To look at the top 10 words that are most associated with each topic, we re-run the model specifying 5 topics, and use show_topics. You can use a simple print statement instead, but pprint makes things easier to read.
ldamallet = LdaMallet(mallet_path, corpus=corpus, num_topics=5, id2word=id2word, random_seed = 77)# Show Topicspprint(ldamallet.show_topics(formatted=False))
Below are the top words for each topic, and my labels for them. Most of the topics are related to the gameplay, except the last one. Since we only looked at negative reviews, we can assume that these are problem areas.
Here’s a quick overview of each topic after inspecting the reviews:
Time: Play time was too short. Many players completed the game in 10 to 25 hours
Experience: The story was lacking, and overall game play was too easy. Moreover, the much hyped-up wild area was poorly animated, and mostly empty.
Visuals: Animation, graphics, and models lacked quality and texture. Expectations were sky high on the visuals because Game Freak cited this as the reason for cutting out so many Pokémon from the game.
Content: Players often compared the switch games to the 3DS games, which had a lot more features, content, and available Pokémon.
Developer: Players felt betrayed by Game Freak because it did not deliver on its promise of stunning visuals. They felt Game Freak lied to them, and that the cut in Pokémon was not justified.
LDA is a good way of finding topics within texts, especially when it’s used for exploratory purposes. I think MALLET LDA gave a pretty good overview of why players hated the game. I did a quick exploration of the positive reviews, and two broad topics came up: gameplay and positive feelings. Perhaps, this might be the case of new players who started Pokémon on the switch (and are hence using Let’s Go as a benchmark) versus seasoned players who are more used to complex gameplay.
**Code can be found here**