Generic Review Scraper for any Android App
By: Ashhadul Islam
Tags: Data Visualization, Android, Data Science

Little drops make an ocean

At our workplace, we are always on the lookout to make Data Science easily available to the users. “Productisation of intellectual capability in the form of pipelines” is a motto that we have been rigorously adhering to.

With that goal in mind, we recently open sourced our in house google play store scraper app. If you are an android developer and are interested to see how your app is doing through the user reviews, then this is the app you would love to use.

Different data points in the reviews

The google play store reviews section contains a host of details. The stars given by each user, the date of evaluation and also the review comments given by user. There is also another metric of how many users found the review useful.

It would be great if we could consolidate the information and get it in a single place to enable our analytics on that.

Keeping that in mind, we built an application that scrapes the google play store for any app of your choice, consolidates the reviews into a word cloud and also allows you to download a sheet containing the reviews and other details for deeper analysis.

Here is the link to try the same. As the application is running on a free instance of heroku, it might take some time to start.

When you open the link, it looks like this.

Play Store Scraper Website

In the first field, enter the URL of the app in the google play store.

Copy the URL for your app in play store

Paste the same URL in the first textbox of the application page

Click on Submit to see the magic

Once you click on submit, the application opens the play store and starts scraping the reviews. Finally, a word cloud will be displayed that contains the most frequently used words in the reviews. Of course, common words like “is”, “are”, “and” etc will be removed in advance.

Word Cloud

Now, if you look at the word cloud, you might find some words that don’t really add much value to your analysis. For example, words like app and bank are not useful because we know we are reviewing an android app and the app belongs to a bank.

Also the name of our bank is DBS, so the word dbs also is inconsequential.

In order to remove those words from the cloud, we now shift our focus to the second text box in the form.

Second field allows removal of domain related stop words

In the second field, you can mention words like app, bank, dbs and hit submit again. This time, processing will be faster as the data set containing reviews is retained for a week.

Added words to be removed. These words are different for different apps

When you click on submit, this time the unwanted words will not appear and you can get a better understanding of the performance of your app.

Makes more sense now

As you can see, there seems to be some issues with the “debit” “card”. Let us go to the play store and check.

22 mentions of card

There seems to be a recent issue with the debit card. On going through the play store reviews, we found that people are facing issues with the activation of the pin of the debit cards issued by DBS.

Thus, besides giving a general view on the performance of your app, you can even get to know about any recent outages or issues faced by your users.

Also, the reviews can be directly downloaded by hitting the Get CSV Button.

Click on Get CSV to get the excel sheet containing reviews and other details

The excel file contains all the details like, the name of reviewer, the date of review, the number of stars given and also the count of people who found the review helpful. Needless to say, it also contains the review text given by each user.

An average of 75 reviews

Now you have a word cloud as well as a consolidated sheet containing all the recent reviews.

Feel free to use this app and check the source code here.

If you face any issues or if you feel that there is a feature that the app absolutely cannot live without, please feel free to create an issue/feature request in github.

Inspiration partner:

“It is every man’s obligation to put back into the world at least the equivalent of what he takes out of it.” — Albert Einstein