The Fraud Squad
By: Raghunandh Gs
Tags: Fraud, Product Management, Data, Retention, Data Science

All but one

How many facebook IDs do you have? Yes, including the one you created just to stalk on a couple of your acquaintances who blocked you and the one you created faking your gender to play a prank on your friend. Maybe three? Maybe four?

In our app, we have players who have more than 20 email ids. No that’s not just one guy. Its a bunch of users who have upwards of over 20 Facebook IDs. Surprised? We were too, when we first found it out. Welcome to the world Chaalbaaz, our skill gaming platform.

This blog will be a journey through the lens of data which will focus on a type of fraud that was happening in our app. It not only wasn’t in the spirit of the game but also affected our core metrics.

Play by The Rules

In order to understand the fraud that was happening you have to first understand the game. You need to know the rules to break the rules.

Chaalbaaz has multiplayer quiz tournaments in Hindi language. We were running two slots one between 1 PM to 3 PM and another between 8 PM to 11 PM. In these slots, users can come, initiate a quiz that is active for that slot and gets matched with other users who have initiated a quiz. The quiz is usually a set of 10 multiple choice questions and the users are supposed to discuss and pick an option. Based on the number of right answers and the time taken to complete, each user will be rewarded with a small of amount in-app cash at the end of the quiz. This in-app cash can be redeemed by the users once they reach a certain threshold amount of total winnings. One thing to note here is we are not getting in any money from the user but we are giving out small rewards. You can call it free cutting chai, if not free lunch to go by the reward amount.

Since money is involved there is a cap on the number of quiz attempts you can make in a slot. Each user, whom we identify by their Facebook log in can participate and complete one quiz per slot. They are not allowed to participate in another quiz in the same slot if they have already completed a quiz or dropped off midway while playing one. However, a player can quit any number of quizzes and start another one provided he quits before he answers the first question. Since this game required a collaborative effort from two players, we thought they must be given a chance to get matched with other users if they feel the opponent whom they have got matched with the first one or two attempts turn out to be lousy guys.

Wait. What?

Being a data-informed org, we monitor most of the metrics which we think are important to monitor on a daily basis. On one such morning, musing over adding more features and listening to music , one of our PMs observed that the Game complete ratio was going down now for a while.

Ah! doesn’t it look like a slide in the park near your home? What a sight for a PMs to look at! These numbers were stooping below 40% in the range of 30% which implies only one-third of all the games started ends in a complete. All these days these numbers were fine, there is no reason why they are going for a nose dive. We were breaking our heads trying to figure out what is stopping our users from completing games. We decided to dig a layer beneath this monitoring metrics.

What? Wait.

The first thing out of the bunch of things we looked at was the distribution of the number of starts per user in a day in the form of a histogram.

The first thing that took us by a shock was the fact that there were a considerable amount of folks starting a quiz more than 100 times a day. Yes, starts. Which means they were matched to another player to play a quiz, the quiz started and they decided to quit. They were quitting more than 100 times a day. Okay. Once or twice in a day, you might get matched with someone whom you don’t want to play with, but 100 times? Seriously? Can’t these players cooperate with a player whom he has got matched with for 5 minutes why do you want to quit so many times? We are not running a matrimony app after all its a multiplayer quiz. Then we charted out a histogram of completes, similar to stops.

This single graph took everyone who was part of this app including the engineers who have seen a lot more shittier things in the past by a storm. Ideally, there were two slots per day and a user will be allowed to play only once per slot. Out of a bunch of unique Identification numbers, we identify each user mainly using two ids. One is by their Device ID(which is unique to each device) and another is by their facebook id(Unique to a user). For all the computations related to app performance like DAU, Daily new user, retention, stickiness etc we use a device ID, To uniquely identify a player we used his facebook ID. This is mainly for the sole reason that user might switch his device but we should be able to identify them so that we will able to restore their history, winnings, wallet balance etc. We all expected few players who have dual sim phones to have a couple of facebook ids and we were okay with allowing them to participate in the quiz once more in the same slot. Even then they must be able to complete only 4 quizzes per day but we have the number of completes ranging more than 10 times that value.

We found that people were using multiple facebook logins to make money sitting in the same place. If x is the maximum money a player can make in a day they were making 10x of that with ease without spending a rupee.

The Trouble Bubble.

We schedule a bunch of new 10 new questions per each quiz. The reward was obviously calculated on the number of correct answers and time taken to answers. Since these guys were playing the same quiz over and over again the got better with each attempt and with each of their ids they will easily win 3x 4x times more reward than an average user. One reason why they quit so many times before starting a quiz is that they have the habit of assessing their opponents.

As they knew most of the answers already they just wanted their opponents to listen to them and press the answer they say. If they don’t align, they will be quit upon. So put in a nutshell, these guys found a way to make money at ease, they were spoiling the user experience of other users and to add to all that since we were calculating all the app metrics on the device IDs they did not help us boost those numbers too. Meaning, even though they were using 20 plus ids and making 10x more than what is allowed, since all those were from a single device they will be counted only as one active user. How sad! We decided we definitely have to teach them a lesson and such behaviour should not be encouraged at all. In order to get the motivation to fight the evil we sat and watched three Rohit Shetty films back to back and took a few decisions which backfired.

Bursting the Bubble and Backfire.

As an effect of following what’s happening around the country every day the first thing that struck our head to fight this evil was to Ban. We decided to ban users who are involved in such fraudulent activities. We ran a query, found all the fraudsters got all their facebook Ids and blacklisted them so that they cannot participate in any event. As soon as we blocked them the amount of fraud happening went down considerably, some of those guys even mailed to our customer support desk saying that they were not able to play the game. We were a little relaxed and had a CID look on our face for spotting the fraud, fighting and defeating it. But it lasted only for a couple of days as the fraudsters found a way to get around it by creating new facebook IDs.

Our next move was to reduce the slot timings from 2 hrs in the morning and 3 hrs in the evening to 6 slots of 15 minutes each at various times of the day. Even though there were multiple reasons to go for 6 slots a day which will run of 15 minutes each, minimising fraud was one of the prime reason. As we were running shorter slots we thought now fraudsters don’t have much time to fool around and that will bring the fraud down as well as improve the complete to start ratio. We ran this for 5 days and in turn what we expected happened the fraud numbers went down and the completion to start ratio went up.

But something which we did not expect also happened. That is D1 Retention numbers went down. As most of our new users were organic we did not have control on the time they install the app. As it happens randomly, there are many chances that there is no quiz scheduled when the user opens the app for the first time. This was the case previously too but we had covered 5 hours of active quiz time out of 24 hours but now it is only 1.5 out of 24 hours (15 mins*6 slots). Users come back the next day(D1) after installation based on the experience they had on their installation day (D0). Forget giving a good experience or bad experience on their first day, due to the reduction of these slot timings we were able to give only a bunch of new users something called experience. And Of course, only a part of it returned next day which brought down our D1 numbers.

Damage Limitation

Since more than other things retention was our at most priority we decided to revert back to the old model of 2 slots. So this time while reverting back to the old model, we added a cap of a total number of quits allowed per id per slot to two. If you quit more than twice you wouldn’t be allowed to participate in that quiz again. However, this had nothing which can tackle fraud, we thought I will at least prevent the fraudsters quitting multiple times in the same slot on multiple different times which spoils the user experience of a noble player. This cap helped in improving the complete to start ratio as well as reduce the touch points of fraudsters. This has reduced the impact of fraudsters on normal DAU users by 70% — which in turn helped in improving the retention.

Not so perfect World

While we continue to fight frauds, they continue to find flaws. But thanks to the data and metrics — we can always dig deeper. While we haven’t figured out the perfect solution to prevent fraud we are focusing on minimising it with every release without impacting other metrics. We realised that we don’t live in a perfect world where every fight good triumphs over evil. After all, this is a game and not a Rohit Shetty film.