Creating a Web Application to Analyse Dengue Cases | by Benedict Soh | Towards Data Science

Information is power. Information on your fingertips is superpower.
During this period, I chanced upon many visualisations on COVID-19 on Reddit, Twitter, and Facebook. Many of them are very inspirational and it is so pleasing to be able to ingest a huge amount of information in such a short period of time.
I was motivated to build a dashboard as well on COVID-19 but I felt that there are more than enough out there. Hence, I decided to work on another issue, Dengue Fever, which is also a pressing issue. Data is readily available on data.gov.sg, but there are 3 main issues:
Some data does not have any visualisationsData is all over the place and additional effort is required to filter the resultsThere is not enough data to tell a story
Some data does not have any visualisations
Data is all over the place and additional effort is required to filter the results
There is not enough data to tell a story
This drove me to build a dashboard on my own. There are 3 main parts to this project:
Analysis: I visualised the number of dengue cases, and created an overlayed map of dengue hot zones and breeding spots.News retrieval: This tab is able to retrieve news from multiple sources on the topic “Dengue”.Twitter Analysis: This feature forms the main bulk of the project. In short, I retrieve twitter data in real-time and analyse their sentiments, presenting them in charts.
Analysis: I visualised the number of dengue cases, and created an overlayed map of dengue hot zones and breeding spots.
News retrieval: This tab is able to retrieve news from multiple sources on the topic “Dengue”.
Twitter Analysis: This feature forms the main bulk of the project. In short, I retrieve twitter data in real-time and analyse their sentiments, presenting them in charts.
The purpose of this project is to show how simple it can be to build such a powerful application. It need not be on Dengue. It can be on any topic out there. For this project, it is not the content that matters, but the skills. This is more of a proof of concept rather than a final product. I will be walking you through all the 3 different parts of the project while providing snippets of code. Hopefully, you will be able to replicate it for your own meaningful projects. Do scroll to the bottom of this article for the link to the full code.
Before I start, I would like to talk about how this project is built.
This project is built with Dash, which is a Python framework for building Data Science/ML projects. Along with dash bootstrap component (DBC) and dash core components (DCC), you will need minimal HTML/CSS skills to build your web application.
Each of the 3 features taps on different technologies (will be further explained). All these features are then deployed using Heroku, which is a cloud platform as a service (PaaS), that allows you to deploy your application to the cloud with ease.
Dash applications mostly consist of 2 sections.
App layout is the front-end portion of your application. However, all thanks to DBC and DCC, you do not need complex HTML codes to create your application. For instance, my entire app was created with the following code:
# Code partially hidden for readability# Layout of entire appapp.layout = html.Div(    [        navbar,        dbc.Tabs(            [                dbc.Tab(analysisTab, id="label_tab1", label="Visuals"),                dbc.Tab(newsTab, ...),                dbc.Tab(socialMediaTab, ...),                dbc.Tab(infoTab, ...),            ],            style={"font-size": 20, "background-color": "#b9d9eb"},        ),    ])
Of course, you will have to specify what is analysisTab, newsTab, etc. You can see how abstract your application becomes with DBC and DCC. Both DBC and DCC provides buttons, tabs, forms, navigation bar along with many other essential components.
App callbacks can be considered the backend portion of your application. If you were to observe the above code, you will notice that I assigned an id="label_tab1" to my analysisTab this helps us to identify which callbacks to use for that specific component.
An example of a callback is as shown:
@app.callback(    Output("sa-graph", "children"), [Input("interval-component-slow", "n_intervals")])def sa_line(n):    children = [...]    return children
What this does is that it will take an input component interval-component-slow with a specified parameter n_intervals and output the results children that was created by a method sa_line() to the output component sa-graph. This is especially useful if you want to make your application interactive. For instance, you can allow your visualisations to change according to a radio button that was clicked by a user.
However, a callback is not always required if you want your application to be static. You will realise that my first 2 tabs did not make use of callbacks as I do not require any inputs from my users. I am only required to resent information to them. The 3rd tab (twitter analysis) is a special case. I had to use callbacks as I want my analysis to refresh every 30 seconds. More about this later!
Full code here.
The main learning point for this section is the usage of Geopandas and Folium.
The data downloaded from data.gov.sg comes in a GeoJSON format. GeoJSON files allow you to encode geographical data into an object. It has a very structured format which includes the type, geometry, and properties of the location. Since dengue clusters involve a plot of land, our locations are encoded in a Polygon object, which is an array of longitudes and latitudes.
Reading a GeoJSON file is easy. We simple use Geopandas:
central_data = gpd.read_file("data/dengue-cases-central-geojson.geojson")
Once the data is loaded, manipulation should be pretty easy, provided that you are well versed with a DataFrame.
Folium allows us to visualise our GeoJSON data in a leaflet (a JavaScript library for interactive maps) map.
First, we create a map and zoom in to our desired spot (in my case, I chose the longitude and latitude of Singapore):
kw = {"location": [1.3521, 103.8198], "zoom_start": 12}m = folium.Map(**kw)
Next, I add 2 layers (one for clusters and another for breeding zones) to the map:
folium.GeoJson(<FIRST LAYER DATA>, ...).add_to(m)folium.GeoJson(<SECOND LAYER DATA>, ...).add_to(m)
Lastly, since Dash does not allow me to display folium objects directly, I exported the resulting map into an HTML file and displayed that using an IFrame:
m.save("dengue-cluster.html")html.Iframe(id="dengue-map", srcDoc=open("dengue-cluster.html", "r").read(), ...)
Full code here.
The main learning point for this section is the usage of the NewsAPI. NewsAPI allows us to search for news articles from multiple sources using their API. Before you can start using it, you will be required to sign up for an account in order to obtain your own unique API key.
Note that as a free-tier member, you will be limited in terms of the number of news retrievable per day.
Keys are very important! Just like how you would not share your house keys with your friends, you will not want to share your API keys. When you are developing on your local system, I highly suggest that you keep all your keys in a keys.py file and add that file to .gitignore so that it does not get pushed online.
Since you did not push keys.py to GitHub, Heroku cannot access it. Instead, Heroku have their own configuration variables. You can add your keys to Heroku either via the CLI or their GUI.
For deployment purposes, we get our code to retrieve keys depending if we are on our local system or Heroku:
try:    from keys import newsapikey  # retrieve from local system    newsapi = NewsApiClient(api_key=newsapikey)except:    newsapikey = os.environ["newapi_key"]  # retrieve from Heroku    newsapi = NewsApiClient(api_key=newsapikey)
Next, we can finally retrieve our news articles:
all_articles = newsapi.get_everything(        q="dengue singapore",        from_param=date_n_days_ago,        to=date_now,        language="en",        sort_by="publishedAt",        page_size=100,    )
There are 3 main functions newsapi.get_everything() , newsapi.get_top_headlines() and newsapi.get_sources(). Use any you deem fit! We can then retrieve our content. An example of retrieving the title of all articles is as shown:
all_articles_title = [        str(all_articles["articles"][i]["title"])        for i in range(len(all_articles["articles"]))    ]
Now that you have your data, you can add them using DBC. For this project, I made use of CardImg, CardBody, and CardFooter.
This forms the bulk of the project. I made use of Tweepy, which is a Python library that allows us to access the Twitter API to stream tweets. My aim is to retrieve all tweets related to ‘Dengue’, pre-process the text, analyse the sentiments, and visualise the results. Tweets are continuously being mined from Twitter and my application will refresh every 30 seconds to display the new data.
At this stage, I should probably introduce to you the workflow.
Tweets are being pulled from TwitterTweets are being cleaned (translated to English, emojis removed, links removed) and sentiment of the tweets are analysed by TextBlob)A connection to a database is established and the processed tweets and sentiments are pushed to the database every time a new tweet comes in. For a local system, I used MySQL and for Heroku, I used their PostgreSQL database. This script is kept running.At the same time, I retrieve tweets that are older than a day and delete them from the database. This will ensure that our database does not run out of space.With my application running in parallel, it will retrieve an hour's worth of data from the database every 30 seconds and display them.
Tweets are being pulled from Twitter
Tweets are being cleaned (translated to English, emojis removed, links removed) and sentiment of the tweets are analysed by TextBlob)
A connection to a database is established and the processed tweets and sentiments are pushed to the database every time a new tweet comes in. For a local system, I used MySQL and for Heroku, I used their PostgreSQL database. This script is kept running.
At the same time, I retrieve tweets that are older than a day and delete them from the database. This will ensure that our database does not run out of space.
With my application running in parallel, it will retrieve an hour's worth of data from the database every 30 seconds and display them.
Just like NewsAPI, Tweepy requires us to generate API keys as well. Head on to Twitter Developer to get yours!
Full code here. Note that commented codes are for MySQL and uncommented codes are for PostgreSQL.
Step 1: Initialise the database by creating a database for us if the database and table do not exist.
DATABASE_URL = os.environ["DATABASE_URL"]conn = psycopg2.connect(DATABASE_URL, sslmode="require")cur = conn.cursor()cur.execute(    """        SELECT COUNT(*)        FROM information_schema.tables        WHERE table_name = '{0}'        """.format(        parameters.TABLE_NAME    ))if cur.fetchone()[0] == 0:    cur.execute(        "CREATE TABLE {} ({});".format(            parameters.TABLE_NAME, parameters.TABLE_ATTRIBUTES        )    )    conn.commit()cur.close()
Step 2: Initialise Twitter API and start streaming
myStreamListener = MyStreamListener()myStream = tweepy.Stream(auth=api.auth, listener=myStreamListener)myStream.filter(track=parameters.TRACK_WORDS)
Step 3: Process every tweet coming in. In order to achieve this, you have to override tweepy.StreamListener, which is the class responsible for pushing new tweets from Twitter to you.
# Override tweepy.StreamListener to add logic to on_statusclass MyStreamListener(tweepy.StreamListener):    def on_status(self, status):        # Start modifying        id_str = status.id_str        created_at = status.created_at        clean_text, text = preprocess(status.text)        sentiment = TextBlob(text).sentiment        polarity = sentiment.polarity        ...        # SQL statement for pushing tweets to DB        ...        # SQL statement for deleting and cleaning DB        ...
The trick to getting your application refresh every x seconds is with dcc.Interval. By adding an interval to your application as an input to your callback, you are telling your application to regenerate your component every x seconds.
dcc.Interval(            id="interval-component-slow",            interval=30000,  # in milliseconds            n_intervals=0,        ),
Full code for visualisations here.
Firstly, we have to retrieve the data that was scraped:
# Loading data from Heroku PostgreSQL    DATABASE_URL = os.environ["DATABASE_URL"]    db_connection = psycopg2.connect(DATABASE_URL, sslmode="require")# Load last 1 hour data from MySQL    ...query = "SELECT ... FROM {} WHERE created_at >= '{}' ".format(        "dengue", timenow    )df = pd.read_sql(query, con=db_connection)# Convert UTC into SGT    ...
Next, the data is cleaned and transformed for the plotting of a time series:
# Clean and transform data to enable time series    # Bin into 5 minutes    result = (        df.groupby([pd.Grouper(key="created_at", freq="5min"), "polarity"])        .count()        .reset_index()    )    ...
Lastly, with the results, we visualise them using Plotly Graph Objects. The line graph on the top left is created with go.Scatter while the pie chart on the top right is created with go.Pie. That’s how easy it is to create your visuals!
Tweets are displayed with Dash DataTable.
dash_table.DataTable(id='table', columns=[{"name": i, "id": i} for i in df.columns], data=df.to_dict('records'))
There currently isn’t much support for word clouds. The conventional method is to create an image and append it to our visualisations. The unconventional method which I used, plots the words on an X and Y axis and instead of displaying the coordinates, I display the text. Hence, this word cloud is created using go.Scatter.
go.Scatter(x=wc_data.x,           y=wc_data.y,           mode="text",           text=wc_data.words,           ...)
Now that your application works well in your local system, it is time to deploy it to Heroku. I highly recommend you check out this 15 minutes tutorial on how to do so. Otherwise, refer to the summary below:
Create a Heroku account and download Heroku CLIEnsure that all dependencies for your project are downloaded. Additionally, run pip install gunicornCreate a Procfile which will tell Heroku what to runCreate requirements.txt with pip freeze > requirements.txt. This is essential for Heroku to understand which packages you are using so that they can download it as well.Deploy your application with
Create a Heroku account and download Heroku CLI
Ensure that all dependencies for your project are downloaded. Additionally, run pip install gunicorn
Create a Procfile which will tell Heroku what to run
Create requirements.txt with pip freeze > requirements.txt. This is essential for Heroku to understand which packages you are using so that they can download it as well.
Deploy your application with
heroku loginheroku create "<YOUR APP NAME>"git add .git commit -m "<ANY MESSAGE>"git push heroku masterheroku ps:scale web=1
Note: for your tweet scrapper and application to run at the same time, you will need 2 different applications. Otherwise, you can pay for additional dynos, which are Heroku containers. Your Procfile will be different. To utilise the same database, set the DATABASE_URL (config vars) of your scrapper to be the same as the DATABASE_URL of your application.
Also, do note that it will take quite some time for the application to load. This is because I am using free dynos. Free dynos will be put to sleep if the application is not accessed for a set amount of time.
This application is simple, lightweight, and powerful. It is by no means a perfect application due to the limited budget, time, and web design skills. I have also yet to take the time to optimise the algorithm, which should greatly increase the speed at which the application loads.
With this article, I hope that you have gained a deeper understanding of Dash, Tweepy, NewsAPI, Geopandas, and Folium. I highly implore you to create similar applications for different contexts.