SQL Case Study: Investigating a Drop in User Engagement | by Terence Shin | Towards Data Science

ProblemBrainstorming Potential CausesDataAnalysis: Event-level AggregationAnalysis: User-level AggregationAnalysis: Engagement by GeographyAnalysis: Email activity and CTRsAnalysis: Clickthrough rates by deviceAnalysis: Weekly digest vs re-engagement emailsSummaryRecommendationFurther Steps
Problem
Brainstorming Potential Causes
Data
Analysis: Event-level Aggregation
Analysis: User-level Aggregation
Analysis: Engagement by Geography
Analysis: Email activity and CTRs
Analysis: Clickthrough rates by device
Analysis: Weekly digest vs re-engagement emails
Summary
Recommendation
Further Steps
This case can be found here if you would like to try it yourself or follow along.
Yammer is a social network for communicating with coworkers. Individuals share documents, updates, and ideas by posting them in groups. Yammer is free to use indefinitely, but companies must pay license fees if they want access to administrative controls, including integration with user management systems like ActiveDirectory.
Yammer noticed a drop in Weekly Active Users, “the number of users who logged at least one engagement event during the week starting on that date”. Engagement is defined as having made some type of server call by interacting with the product (shown in the data as events of type “engagement”.
My goal is to determine the cause of the dip in the image below.
Potential causes of a dip in engagement include the following:
Product-related issues. There may have been a bug in the product that deterred users from using the product. Similarly, Yammer may have released a product update around the beginning of August, which a significant number of users did not like.
Time of the year. August may simply be a month when a lot of users go on vacation. This may be the case if a significant portion of users are parents, as they tend to go on vacations before September when school starts.
Competitors. It’s possible that a competitor launched a competing product at the beginning of August, or a competitor released significant updates to an existing product that caused users to churn.
Data is not being tracked properly. It’s possible that the data pipelines are not working, and thus, the data is not being tracked properly. This is a problem caused internally and would require the help of data engineers to fix.
Marketing event. When large promotions are launched, it’s normal that a company will attract lower quality customers because of the discounts provided. Therefore, it’s a possibility that the dip in August is the aftereffect of a large marketing event in July.
Each row represents one user, with information associated with each user.
Each row represents an event, which is an action that a user has taken on Yammer. These events include login events, messaging events, search events, events logged as users progress through a signup funnel, and events around received emails.
This table contains data on events related to sent emails. It is similar in structure to the events table above.
The final table is a lookup table that is used to create rolling time periods.
The first thing I wanted to see was what events are categorized as ‘engagement’.
SELECT DISTINCT event_name, event_typeFROM tutorial.yammer_eventsORDER BY event_type, event_name ASC
The events categorized as ‘engagement’ are as follows:
home_page
like_message
login
search_autocomplete
search_click_result_1
search_click_result_2
search_click_result_3
search_click_result_4
search_click_result_5
search_click_result_6
search_click_result_7
search_click_result_8
search_click_result_9
search_click_result_10
search_run
send_message
view_inbox
with one as (SELECT   EXTRACT('month' FROM occurred_at) as month,  count(event_name) as event_countFROM tutorial.yammer_eventsGROUP BY  month)SELECT   *,  (event_count - LAG(event_count) OVER (ORDER BY month ASC)) as diffFROM one
Using the query above, we can deduce that the difference in engagements between July and August is 18,037.
The next thing I wanted to do was count the number of occurrences of each ‘engagement’ event month over month, to see if there’s any significant event(s) that caused the dip after July 28.
with two as (with one as (SELECT   CONCAT( EXTRACT('month' FROM occurred_at), '-', EXTRACT('year' FROM occurred_at)) as month_year,  event_name,  count(event_name) as event_countFROM tutorial.yammer_eventsWHERE event_type = 'engagement'GROUP BY  event_name,  month_year)SELECT   *,  CASE    WHEN month_year = '5-2014'      THEN 0    WHEN month_year != '5-2014'     THEN (event_count - LAG(event_count) OVER (ORDER BY event_name ASC, month_year ASC))    ELSE NULL END AS abs_changeFROM one)SELECT *FROM twoWHERE  month_year = '8-2014'  AND abs_change < 0ORDER BY abs_change asc
The dip in engagement was largely attributed to home_page, like_message_ view_inbox, send_message, and login. It seems like the drop in all of these events are simply related to the fact that users are logging in less.
Next, I wanted to analyze the data on a user level to see if the drop was due to a drop in users or a drop in the number of engagements/user.
SELECT  EXTRACT('month' from occurred_at) as month,  count(DISTINCT user_id) as num_usersFROM tutorial.yammer_eventsWHERE event_type = 'engagement'GROUP BY month
Using the code above, I found the number of distinct active users on a monthly basis.
We can see that the number of users declined from 3058 in July to 2795 in August, a drop of 8.6%. This is likely the cause of the decline in engagement events, specifically homepage, like_message, view_inbox, send_message, and login.
I also looked to see if the engagement per user declined as well.
SELECT  EXTRACT('month' from occurred_at) as month,  count(event_name) as num_events,  count(DISTINCT user_id) as num_users,  count(event_name)/count(DISTINCT user_id) as events_per_userFROM tutorial.yammer_eventsWHERE event_type = 'engagement'GROUP BY month
The number of events per user fell from 30 to 26, representing a drop in engagement by 13%, which is also a fairly significant number.
Therefore, we need to find out why there was a drop in active users and also a drop in engagement per user.
SELECT  action,  EXTRACT('month' FROM occurred_at) AS month,  count(action) as num_emailsFROM tutorial.yammer_emailsGROUP BYaction, monthORDER BYaction, month
I wanted to take an aggregated look at the email activity to see if there was a change in emails sent, click-through rates (CTRs), or something else that may have caused a reduction inactive users.
Immediately, I noticed that there was a steady increase in the number of emails sent (weekly digest and re-engagement) and emails opened, but there was a decrease in clickthrough rates. From July to August, there was a 6.5% increase in emails open but a 27% decline in clickthrough rates.
I wanted to take a deeper look into the clickthrough rates and see if the decline had anything to do with the devices. It could have had to do with the type of operating system (IOS vs Android) or mobile vs desktop.
with emails as(SELECT   *,  CONCAT(EXTRACT('day' FROM occurred_at), '-', EXTRACT('month' FROM occurred_at), '-',  EXTRACT('year' FROM occurred_at)) as date,  EXTRACT('month' FROM occurred_at) as monthFROM tutorial.yammer_emails emails), events as (  SELECT DISTINCT     user_id,    CONCAT(EXTRACT('day' FROM occurred_at), '-', EXTRACT('month' FROM occurred_at), '-',  EXTRACT('year' FROM occurred_at)) as date,    device,    EXTRACT('month' FROM occurred_at) as month  FROM tutorial.yammer_events  ORDER BY user_id ASC)SELECT   device,  emails.month,  count(emails.user_id)FROM emailsLEFT JOIN events ON  emails.user_id = events.user_id  AND emails.date = events.dateWHERE action = 'email_clickthrough'GROUP BY device, emails.month
Using the query above, I noticed that the clickthrough rates on laptops and computers were stable from July to August, but not the tablets and cellphones.
By categorizing the device names into ‘mobile’, ‘tablet’, and ‘laptop’, I could determine if this is truly the case.
with emails as(SELECT   *,  CONCAT(EXTRACT('day' FROM occurred_at), '-', EXTRACT('month' FROM occurred_at), '-',  EXTRACT('year' FROM occurred_at)) as date,  EXTRACT('month' FROM occurred_at) as monthFROM tutorial.yammer_emails emails), events as (  SELECT DISTINCT     user_id,    CONCAT(EXTRACT('day' FROM occurred_at), '-', EXTRACT('month' FROM occurred_at), '-',  EXTRACT('year' FROM occurred_at)) as date,    device,    EXTRACT('month' FROM occurred_at) as month  FROM tutorial.yammer_events  ORDER BY user_id ASC)SELECT   CASE    WHEN device IN ('amazon fire phone', 'nexus 10', 'iphone 5', 'nexus 7', 'iphone 5s', 'nexus 5', 'htc one', 'iphone 4s', 'samsung galaxy note', 'nokia lumia 635', 'samsung galaxy s4') THEN 'mobile'    WHEN device IN ('ipad mini', 'samsung galaxy tablet', 'kindle fire', 'ipad air') THEN 'tablet_ipad'    WHEN device IN ('dell inspiron desktop', 'macbook pro', 'asus chromebook', 'windows surface', 'macbook air', 'lenovo thinkpad', 'mac mini', 'acer aspire desktop', 'acer aspire notebook', 'dell inspiron notebook', 'hp pavilion desktop') THEN 'laptop_comp'    ELSE null end as device_type,  emails.month,  count(emails.user_id)FROM emailsLEFT JOIN events ON  emails.user_id = events.user_id  AND emails.date = events.dateWHERE action = 'email_clickthrough'GROUP BY device_type, emails.month
Indeed, it seems to be the case that the drop in clickthrough rates was attributed specifically to mobile devices and tablets.
So far, I’ve determined that the lack of engagement is due to a decrease in email clickthrough rates from July to August. To gather more information, I want to see if we can narrow the problem even further by email type.
with one as (SELECT   *,  EXTRACT('month' from occurred_at) as month,  CASE WHEN (LEAD(action, 1) OVER (PARTITION BY user_id ORDER BY occurred_at ASC)) = 'email_open' THEN 1 ELSE 0 END AS opened_email,  CASE WHEN (LEAD(action, 2) OVER (PARTITION BY user_id ORDER BY occurred_at ASC)) = 'email_clickthrough' THEN 1 ELSE 0 END AS clicked_emailFROM  tutorial.yammer_emails)SELECT   action,  month,  count(action),  sum(opened_email) as num_open,  sum(clicked_email) as num_clickedFROM  oneWHERE action in ('sent_weekly_digest','sent_reengagement_email')GROUP BY  action,  monthORDER BY  action,  month
It’s evident that the drop in clicks is due to the weekly digest email and not the re-engagement email.
Through my analysis, I can conclude the following:
The drop in engagement was mainly attributed to a drop in five engagement events (home_page, like_message_ view_inbox, send_message, and login).
I then found that the decrease in events was caused by a reduction in total active users MoM, as well as a decrease in engagement per user.
After I took an aggregated look at the emails table and I noticed that there was a significant decrease in click-through rates from July to August even though there was an increase in the number of emails opened.
By segmenting the clickthrough rates by device type (mobile, tablet, laptop), I noticed that the drop in clickthrough rates was attributed to mobile and tablet devices.
Lastly, the decline in click-through rates is attributed to the weekly digest email and not the re-engagement email.
My recommendation is to immediately take a deeper look into the weekly digest emails specifically for mobile devices and tablets. It’s possible that there’s a technical problem, making it difficult for users to click the email or simply a UX problem, where the content and layout of the email are not enticing users to click. A good first step would be to see what changes have been made from July to August and working backward.
Some other things that could’ve been looked at include the following:
Check to see if the change is attributed to a small number of users (Pareo principle)
Cohort analysis to see if the cause is due to a short user lifecycle
Analysis by language
Analysis by geography
Founder of ShinTwin | Let’s connect on LinkedIn