Data science classification for mobile app malware | Towards Data Science

TL;DR: Bad guys abuse permissions and outdated software to infect your devices.
At the time of writing this piece, Apple Inc. and Epic Games, Inc. are in the throes of a legal dispute. Epic Games claims that Apple’s App Store is a monopoly, and should not have control over in-app purchases. Apple countersued Epic and accused it of breaching its contract. The outcome of the ruling could fundamentally alter how Apple runs its App Store and how app developers monetize their content.
There is one aspect to the lawsuit that piqued my interest: Apple’s argument for maintaining its current model of having complete control over the App Store.
The argument goes that Apple’s tight grip over the App Store guarantees a safe environment where every single application is vetted for malware and compliance with Apple’s standards. Hence, the user is shielded from malware and can focus on what’s important: finding an app you like. The metaphor of the walled garden is often used here since you can pick and enjoy any fruit within the garden as long as you stick within the confines of its walls.
This got me asking: What’s the security landscape in a third party store compared to Apple’s? Can we create a simple data-driven model to vet apps for malware regardless of who controls the store?
Full disclosure: This entire project was done on an Apple MacBook, I’m not taking any sides in the legal case.
My objective in this piece is to show how a data-driven approach can flag malicious smartphone apps, and provide an explanation for each step leading up to that goal. I will also see how my results compare to other research on the topic.
The operating system (OS) examined is Android, which has a 72% share of the world wide mobile OS market. Given its widespread popularity, it is more likely to be abused by malicious actors.
The repository associated with this project can be found here:
github.com
The data is from Kaggle user Saurabh Shahane, and can be downloaded in one click:
www.kaggle.com
The dataset was originally used in the paper “ADROIT: Android malware detection using meta-information” by Martín, Alejandro; Calleja, Alejandro; Menéndez, Héctor D.; Tapiador, Juan; Camacho, David.
Each row in the dataset represents meta-information of an app available on the Aptoide app store website and also information from the app’s Android Manifest. In addition, the paper will serve as a benchmark for model performance.
Please note that I assume that the information in the dataset is as truthful and accurate as possible. Any changes to this fundamental assumption might entirely alter the findings of this article. In addition, I highly recommend reading the paper as it explains how the data was aggregated and labeled.
In this section, I give a general description of how I munge disparate data types into actionable features.
Rather straightforward, the target variable takes one of two values: benign or malware. The data is about 70% benign and 30% malware. For comparison, according to Apple CEO Tim Cook, only 1–2% of the Apple App Store is malware.
The authors of the ADROIT paper achieved 94% AUC. I will see if I can replicate similar results with my methods.
Moreover, I will also use F1 score for checking model performance. Indeed, ROC averages over all possible thresholds whereas the F1 score applies to any arbitrary point on the ROC curve.
Using the F1 score also raises core business questions: do we sacrifice precision for more recall? Is the cost of missing a malware app much higher than barring well-meaning developers from using the marketplace (False Positive)?
To rephrase the previous question: For a store like Aptoide, how much potential income would they be missing out on if they barred a well-meaning developer? On the other hand, how much reputational damage, user harm, and lost income is there from missing malware?
Furthermore, not all malware is created equal. Indeed, it can be adware, spyware, ransomware, or some other kind.
These questions are tied to core business benchmarks and are beyond the scope of this article.
Out of 166 numerical features, 160 are binary and 6 are continuous. We have too many features on our hands, and it might hinder a model’s predictive performance. For this reason, it’s useful to look at each feature’s variance and remove features that have a low variance.
For the continuous variables, we have to look at outliers. This is particularly relevant to the number of ratings an app receives since an overwhelming number of apps never received a single rating.
In order to weed out outliers, I used Tukey’s method: the outliers are values more than 1.5 times the interquartile range from the quartiles, either below Q1 − 1.5IQR, or above Q3 + 1.5IQR.
When it comes to NLP, I usually pose the following questions:
Can I understand what a column is about by skimming the text?
Does it use the Latin alphabet and is it all in the same language?
What are the preprocessing functions needed?
What’s the most compact and representative way to transform the text?
As it turns out, many entries have a description in Chinese characters (Also some Turkish, Korean, etc...). If one day I want to productionize or explain this model, I will have to understand and explain its choices to stakeholders. However, I can’t explain that which I don’t understand and therefore decided to drop non-Latin characters from the text.
Below is a sample from an app description:
Tap Tap Gems is a very addictive puzzle game.  Tap two or more gems with the same colour to remove them from the board.  You must think which gems should be removed next.  To pass each level you must score certain amount of points.   The more gems removed the more points you score.  Game features 120 levels and 7 different gems.
And another sample shows that even with strictly Latin characters, one might still have no idea what is going on (I’m guessing this is Vietnamese?).
Game bi dn gian Vit bao gm 15+ cc mini quen thuc vi ngi Vit ta t xa n nay nh: Xm, Phm, Tin ln min Nam, Ling, Poker, X t, Mu binh, Bi co, X dzach v Tin ln min Nam m l.  t lu Tin Ln Min Nam m L - Game bi dn gian Vit  c cc game th yu qu bi  ha cc p, giao din c chm sc rt k. Cc chc nng trong game hn hn cc game cng loi khc nh
After preprocessing, the text data is transformed using TF-IDF vectorization. In order to optimize prediction, TF-IDF parameters are tuned using a grid search cross validation.
At first I wanted to choose XGBoost since it had the highest AUC. So I set out to do a vanilla data science workflow that comes after data munging:
Instantiate TF-IDF vectorizer
Instantiate a pipeline to avoid leakage
Research how parameters influence the model predictions with GridSearchCV
With an AUC of 0.94, I was happy since it matched the authors’ findings. However, when comparing train/test AUC, the plot below suggested I was overfitting:
In contrast, Random Forest (RF) shows underfitting:
Logistic regression is not overfitting, but it might be severely underfitting.
XGBoost should be the clear winner here, however, I decided to continue with a vanilla logistic regression. It comes with a rather straightforward interpretability and saves me the trouble of using methods such as SHAP, feature permutation, or LIME.
Since this task involves NLP, I want to mine as much information as possible from the text data. Logistic regression tends to suffer when working with sparse data, and sparse data is typical when working with text.
We have two plots below, one is the ROC curve and the other is the precision recall curve. Multiple curves show how as we add more features the model performance improves only slightly.
Spoiler Alert: Text data did not matter, the next section will show that the numerical features in the Android Manifest are the most important factors for classification.
It’s fair to assume that black box models are not exactly popular. Therefore, I set out to understand how the classifier weighs its decisions and dive deeper into these features.
In terms of what is a strong indicator for malware, the top 5 are:
android.permission.READ_SYNC_SETTINGSandroid.permission.WRITE_SYNC_SETTINGSandroid.permission.MANAGE_ACCOUNTScom.android.launcher.permission.UNINSTALL_SHORTCUTandroid.permission.RECEIVE_BOOT_COMPLETED
android.permission.READ_SYNC_SETTINGS
android.permission.WRITE_SYNC_SETTINGS
android.permission.MANAGE_ACCOUNTS
com.android.launcher.permission.UNINSTALL_SHORTCUT
android.permission.RECEIVE_BOOT_COMPLETED
Take for example the permission to write sync settings, what does it mean? When a device synchronizes (syncs), it is synchronizes data on your phone with the service’s servers.
The table below is a snapshot from the training data, and it suggests that giving an unvetted app these permissions is a strong case for caution.
In addition, MANAGE_ACCOUNTS, allows the app to add/remove accounts and delete their passwords.
Moreover, the RECEIVE_BOOT_COMPLETED stood out as a strong indication of compromise. The android documentation sums it up nicely:
Though holding this permission does not have any security implications, it can have a negative impact on the user experience by increasing the amount of time it takes the system to start and allowing applications to have themselves running without the user being aware of them.
Lastly, the UNINSTALL_SHORTCUT permission is no longer supported. This might be associated with malware as older software tends to have more outstanding vulnerabilities.
For the indicators against malware, the main features are not as obvious:
android.permission.BLUETOOTHandroid.permission.READ_LOGS.//Min_SDKandroid.permission.READ_EXTERNAL_STORAGE“ringtones”
android.permission.BLUETOOTH
android.permission.READ_LOGS
.//Min_SDK
android.permission.READ_EXTERNAL_STORAGE
“ringtones”
It’s not entirely clear to me why permissions to connect to paired Bluetooth and reading low-level system log files is a case against malware. Perhaps it’s not as bad as other permissions that allow the device to advertise to nearby Bluetooth devices (see BLUETOOTH_ADVERTISE).
Moreover, it’s beyond me why having “ringtones” in the app description gives you the green light to download it.
However, I can comprehend how setting a higher SDK version (Min_SDK) can protect your device against infection. This feature aligns well with READ_EXTERNAL_STORAGE. The latter is enforced starting API level 19 and prompts the user for themselves to enable/disable the permission.
Unfortunately, the referenced paper does not come with code or an explanation of model weights. Therefore, I’m unable to rigorously compare my findings with the authors’.
Working with data from the Aptoide store suggests that there’s an ample amount of malware on third party stores. Therefore, Apple’s business practices could seem reasonable if they guarantee the safety of the user.
However, data science can be leveraged to create a speedy and accurate classification system where such restrictions by Apple are not necessary. The logistic regression model is simple to set up and offers adequate performance when compared to existing literature.
Projects like these have the potential to boost the safety of app stores and give a deeper understanding of how vulnerabilities are exploited. The methodology can be expanded to other stores and operating systems.
If you’re interested in learning more about the intersection between cybersecurity and data science, check out this other project: