dataesr/scientific-paragraphs-categorization huggingface dataset:
We evaluated the Stacking method using four meta-classifiers: SVM, RF, KNN, and Multi-Layer Perception (MLP), with three hidden layers of 32, 64, and 128 neurons, respectively. The final predictions of this method are given by the meta-classifier. -Stacking Probability, where the prediction probabilities of each classifier are used to train a binary meta-classifier. Similarly, we evaluated Stacking Probability using four meta-classifiers: SVM, RF, KNN, and Multi-Layer Perception (MLP), with the same architecture as in Stacking Prediction. -Ensemble Selection, where the probabilities of each classifier are weighted according to its overall performance on specific metrics (i. e. , F1-score, Recall,. ). Since such a performance must be determined beforehand, we use a validation dataset that serves to iteratively 7  infer the weights for each classifier  (Caruana et al.
2004 ). Experimental results with the described Ensemble Learning techniques are provided in the lower part of Table  4. Again, we provide the detailed scores in Tables  7, 8 , 9, and 10 in the Appendix. We observe that the detection performance still varies significantly across datasets: Min Probability sees the most important variation of 84 percentage points (0. 12 for the whole LITTEMPCONSIST to 0. 96 for 2019 ANDTEMPCONSIST). The difficulty of the whole LITTEMPCONSIST is also confirmed when combining the predictions since the highest F1 score reported in that dataset is 0. 43. The evaluation results for the 16 prediction combination methods show that none of these methods has reported the highest F1 score on all the datasets. Specifically, Ensemble Selection is the best technique in 12 experiments.
Stacking Probability with SVM achieves the highest F1 score in three experiments. As for Max Probabilities, it outperformed the others on one dataset. When Ensemble Selection is not the highest performing classifier, the difference in F1 score between Ensemble Selection and the best method is at most two percentage points. We again conduct the Friedman test to compare the detection performance of the Ensemble Learning classifiers. The test reports a p value of 1. 42 -249 , which confirms that these classifiers do not have the same detection performance. We then proceed with the Nemenyi test and report its results in the sub-figure (c) of Fig. 1. The sub-figure shows that the classifiers used to combine the predictions do not perform similarly. Specifically, the detection performance of Ensemble Selection is not similar to that of all the evaluated classifiers, including Max Probabilities and Stacking Probability with SVM, which have outperformed it on four datasets.
RQ5: Does combining feature sets or predictions from state-of-the-art approaches lead to classifiers that significantly outperform the original detectors? In this section, we aim to compare the detection performance of the state-of-the-art classifiers and the best methods to combine the features and the predictions. The evaluation of the state-of-the-art malware detectors (c. f. , Section 3. 1) showed that no approach has outperformed the others on all the datasets. For example, DREBIN has reported the highest F1 score in nine out of 16 experiments, but other approaches have remarkably outperformed it on the AndroZoo datasets. In Section 3. 3, we have assessed the added value of the merged feature set using six classifiers. Our results showed that Bagging achieved the highest F1 score in 11 out of 16 experiments.
On the DREBIN LITTEMPCONSIST dataset, AdaBoost has outperformed Bagging with 6 percentage points. With the combination of predictions experiments, we have observed the same pattern: No Ensemble Learning method has reported the highest F1 score in all the settings. For example, Ensemble Selection achieved the best detection scores in 12 experiments, but other methods have outperformed it in four evaluation experiments. However, the difference in F1 score between Ensemble Selection and the best approaches in these four experiments is at most two percentage points. Before proceeding with the statistical test, we first compare the detection scores of the best state-of-the-art malware detectors with those reported by the combination methods in RQ3 and RQ4. Specifically, we select from each RQ the combination method that has most often outperformed the others.
For RQ3, we select Bagging as the best classifier trained with the merged feature set. As for RQ4, Ensemble Selection is considered the best method to combine the predictions. We refer to Table  4  to compare the detection performance of these two methods with that of the best state-of-the-art classifiers on each dataset. Overall, Bagging has increased the detection performance in nine experiments. The increase in the F1 score is at most two percentage points except for the whole LITTEMP-CONSIST where it has reached five percentage points. This classifier has also decreased the F1 score in four experiments by one, two, six, and one percentage point, respectively. In the remaining three experiments, Bagging has reported the same detection performance as the best state-of-the-art approaches.
As for Ensemble Selection, it has increased the detection performance by at most two percentage points in 11 experiments. This method has also reported the same detection performance as the best approaches in three experiments and decreased the F1 score by one percentage point in two experiments. Neither Bagging nor Ensemble Selection has remarkably increased the detection performance of state-of-the-art malware detectors. While it has enhanced the F1 score by five percentage points on one dataset, Bagging has also decreased the F1 score by six percentage points on one dataset. For Ensemble Selection, despite improving the F1 score in 11 experiments, this improvement is at most two percentage points. Nevertheless, Ensemble Selection has generally maintained the highest detection performance of state-of-the-art malware detectors independently of the dataset since it has maintained the least performance gap with the best classifiers on all the datasets.
To validate our observations, we conduct the Friedman test on the F1 scores reported by the state-of-the-art classifiers, Bagging and Ensemble Selection. Since the p value of the test is 5. 42-179, we conduct the Nemenyi test and report our results in the sub-figure (d) of Fig. 1. As shown in the sub-figure, the detection performance of both Bagging and Ensemble Selection is different than that of the state-of-the-art classifiers. Moreover, the p value of the test that compares Bagging and Ensemble Selection is greater than 0. 5, which means that we failed to reject the null hypothesis. Our results suggest that there is insufficient evidence to affirm that the detection performance of these two classifiers is different. To sum up, both Bagging and Ensemble Selection have generally maintained the highest detection performance of the state-of-the-art approaches independently of the datasets.
Discussion The literature of Android malware detection lavishes with a huge number of malware classifiers. Each approach aims to capture malware samples by proposing a set of features that is compiled to approximately represent app behaviour. In this study, we consider state-of-the-art approaches published in top venues, and we perform an independent evaluation of their performance. Our evaluation dataset includes a diverse set of apps, spanning across a decade (2010-2020) of app development. Our aim is to challenge the classifiers with diverse samples. We further executed experiments where dataset selections are temporally-consistent (in contrast with typical random sampling), in order to assess malware classifiers' ability to cope with emerging malware. Overall, considering all experimental scenarios, the results show that none of the studied approaches stands out across all settings.
Ensuring high detection performance across datasets Our study shows that malware detectors have significant variability in performance from one dataset to another. Furthermore, no state-of-the-art malware detector could outperform all others in all settings. These results raise questions about the characterisation of the addedvalue of each studied approach as well as its suitability for deployment in production. In an attempt to build a malware classifier that exploits the added-value of all studied classifiers, we have investigated two main approaches: merge of all feature sets and combination of classifiers' predictions. Our experiments show that Bagging and Ensemble Selection have reported promising results: the yielded classifier generally achieves, in all scenarios, a detection score that is as good as the best score reported by individual approaches.
Hypothetic reasons behind the failure of ensemble learning to outperform the state of the art Generally, Ensemble Learning methods aim to enhance the detection performance of the base learners. In our study, however, these methods did not help to outperform the state of the art, although they have provided a detection performance stability across datasets. In the best case, Bagging and Ensemble Selection methods have increased the highest F1 score reported by the base learners by five and two percentage points, respectively. From Table  4 , we observe that there is still room to improve the state of the art, in particular when the experiments are performed in a temporally-consistent manner. Yet, our experiments show that combining feature sets or predictions from these state-ofthe-art classifiers does not lead to the hoped improvement.
Below we enumerate potential reasons why combining the state of the art has not led to a classifier that surpasses all individual approaches: There is a significant overlap of false-negatives in state-of-the-art classifications: We hypothesise that combining state-of-the-art approaches could not enhance the detection performance due to the presence of malware apps that are actually "difficult to detect" for all the approaches. Specifically, the malware that has escaped the detection of the best approaches could not be detected by the other approaches either. To verify our hypothesis, we examine the pairwise overlap of False Negatives (FNs) for the best detector and each of the other detectors considered in RQ1 on average. We provide in Fig. 2 , the distribution of the FNs overlap for the whole datasets.
We also present in Table  6  the number of FNs overlap for the best detector and each of the other classifiers. We observe that there is a significant overlap of FNs in all the datasets. This overlap ranges from 32% in LITTEMPCONSIST to 100% in ANDTEMPCONSIST. These results suggest that malware apps that are "difficult to detect" for the best detector, in a given scenario, are also challenging for the other classifiers: such apps therefore escape the detection despite Ensemble Learning. We also inspect the families of the overlapping FNs from the first data splits in order to identify the major families that are difficult to detect. Specifically, we identify the top five families of the overlapping FNs for the best approach and each detector in the whole datasets. We represent the distribution of the number of malware in the top families in Fig.
3. The results in Fig. 3  show that many overlapping FNs do not belong to a known family (i. e. , "SINGLETON"). We note that the label "SINGLETON", generated by AVCLASS, groups the malware that could not be attributed to any family. The overlap of these samples exceeds 500 malware, which shows that they are indeed difficult to detect. Regarding the known families, we observe that "fakeapp", "jiagu", "secapk", and "dnotua" represent the Fig. 2  The distribution of the average number of overlapping FNs for the best detector and each of the other approaches in the whole datasets Fig. 3  The distribution of the number of overlapping FNs' from the top five families in the whole datasets families with the highest number of overlapping FNs. Overall, inspecting how these samples differ from the other TPs belonging to the same families might help design new approaches that can detect the escaped malware.
Temporally-consistent experiments are challenging: In Table  4 , we observe that the best combination results are achieved in the temporally-inconsistent experiments. Moreover, we have seen in Section 3. 5 that Ensemble Selection has decreased the detection performance of the original classifiers in two experiments which are both temporally-consistent. Furthermore, the individual approaches themselves have somehow reported a poor detection performance especially for the Literature datasets (cf. Section 3. 1). For the whole LITTEMPCONSIST, we have shown that all the state-of-the-art classifiers report F1 scores that are below 0. 5. Since these classifiers make mistakes more often, their combination could not offer any improvements, and has even resulted in a slight decrease of the performance.
We note that our results are in line with previous studies  (Pendlebury et al. 2019; Allix et al. 2015)  in which the authors show that the performance of malware detectors is significantly decreased in a temporally-consistent setting. The limited performance reported on the temporally-consistent experiments can be explained by the evolution of Android malware and the emergence of new malware families. Indeed, Android malware is evolving fast, and new families can exhibit previouslyunknown behaviours. In the temporally-consistent experiments, the test dataset is likely to contain malware belonging to families that were unseen in the training dataset. In the temporally-inconsistent experiments, this situation is possible, but less likely due to the randomness of the split. Given that the training is supposed to characterise maliciousness, if the training set is not representative of the different families, the model will not generalise to samples in the test set, which would lead to poor detection performance.
The diversity of the feature sets is limited: The 10 studied detector variants each leverage a different feature set. In the whole ANDTEMPINCONSIST dataset, we have found that the overall number of features across all studied approaches surpasses 19 Million features. This huge number of features could have suggested that, altogether, state-of-the-art approaches have sufficient information to correctly predict malware apps. Unfortunately, our experiments show that even merging all the features set to train a single machine learning algorithm does not lead to capturing all malware samples. We thus hypothesise that the overall feature set is not more representative (i. e. , does dot capture more relevant information) than individual feature sets proposed by different approaches. It is indeed plausible that the different feature sets are actually redundant across approaches, with respect to malicious behaviour characterisation.
This raises a concern in the literature on the added-value of ever-renewed feature sets using the same types of analyses. Recently, researchers have started to investigate novel ways to represent Android apps  (Daoudi et al. 2021c; Sun et al. 2021; Huang and Kao 2018; Ding et al. 2020). In particular, the feature engineering process, which was largely manual, has been tasked to be resolved via deep learning. To that end, artefacts from the app package (e. g. , the DEX file, Manifest, etc. ) can be processed (e. g. , via image representation) to be fed to neural networks for automatic features extraction. Such alternative features may help improve malicious behaviour characterisation. Classification algorithms leveraged in our study may have limited capabilities: The studied approaches rely on three common classification algorithms: Linear SVM, RF, and KNN.
More sophisticated Ensemble Learning techniques may lead to different results in future work. AV labels used in our study may be noisy  (Hurier et al. 2016; Salem et al. 2021; Xu et al. 2021) : Android malware datasets are generally created using labels from AV engines or online services such as VirusTotal 8. Antivirus engines have been used to label most of the apps in our datasets as malware or benign. Since the AV engines may have different classification decisions, their use can result in a noisy dataset. Researchers usually consider an app as benign when it is not flagged by any antivirus. In order to label an app as malware, a threshold of antivirus agreements needs to be defined. Specifically, the malicious label is attributed to the apps that are detected by a number of antivirus engines that is equal to or above the specified threshold.
Some researchers choose a higher threshold value in order to increase the likelihood that the apps are truly malware. Other researchers prefer to decrease the threshold value so that the "grey" malware are also included in the dataset. While the second strategy can indeed help to learn from the "difficult" malware, it can results in including False Positive labels in the ground-truth dataset. Our experiments use datasets, from different sources, which are compiled using different strategies. This may have introduced noise that is challenging to estimate.
Threats-to-validity The results and findings of our study are subject to some threats to validity. In this section, we enumerate these threats and explain how we attempted to alleviate their impact. First, since the generalisability of our conclusions highly depends on the evaluation dataset, we have considered two large datasets of Android apps to extend the validity of our findings. The literature dataset has been used to evaluate Android malware detectors in the literature, and it includes over 197K apps. We have also removed the duplicated apps in the whole literature dataset to avoid evaluation biases and comply with the recent recommendations about sample duplication  (Zhao et al. 2021). As for the AndroZoo dataset, it contains over 265K samples that we have collected from AndroZoo  (Allix et al.
2016b ) repository. Moreover, apps in our datasets span from 2010 to 2020, which helps to thoroughly assess the performance of the studied approaches. Furthermore, we have also included the Literature and AndroZoo subsets in our evaluation to diversify our settings. Second, we study the possibility of combining state-of-the-art malware detectors. Since Android malware detection literature is prolific, selecting the evaluated subjects is not straightforward. To eliminate any selection bias, papers from 16 major venues in Software Engineering, Security, and Machine Learning have been considered. Third, the implementations of the evaluated approaches might also bias our results. To mitigate this threat, we have considered the approaches that have been reported to be reproducible in the literature  (Daoudi et al.
Assessment of existing work Researchers have started long ago to invest in reviewing existing work on malware detection. A survey  (Rossow et al. 2012 ) has been conducted to assess the methodological rigour and prudence of 36 malware execution papers and has stressed the need for the community to ensure better handling of the datasets. The use of the most recent training labels from VirusTotal has been shown to artificially inflate the detection performance of malware detectors  (Miller et al. 2016). A temporal label consistency constraint has then been introduced to ensure that the training labels are temporally precedent to the evaluation samples. Ten sources of biases have been identified based on the revision of 30 papers from toptier security venues  (Arp et al. 2020 ). These biases can affect the results reported in machine learning based computer and network security research.
2015; Alam and Vuong 2013; Zhang and Jin 2016)  for Android malware detection. For example, a total of 179 static features that include API calls, (Linux/Android) commands, and Permissions are extracted and fed to a Random Forest Ensemble Learner  (Yerima et al. 2015). This same approach is re-used by augmenting the features set with semantics-based features extracted from the sinks and sources flows  (Zhang and Jin 2016). RF has been combined with KNN as base Learners to predict malware using sensitive API Calls from a small dataset of 1044 apps  (Zhao et al. 2018). Probabilities of predictions from RF and KNN have been weighted with 0. 6 and 0. 4 respectively to form the final decision. One year later, the same approach was slightly modified by adding the Permissions to the features set  (Zhao et al.
2019). Other methods of Ensemble Learning have also been evaluated. Stacking refers to training a meta model on the predictions of other based learners. Logistic Regression has been used as a Stacking algorithm to combine the output of Random Forest, SVM, and KNN algorithms that are trained using features from AndroMD dataset 9  (Dhalaria and Gandotra 2020). MuViDA  (Appice et al. 2020 ) is a multi-view malware detection approach that is based on clustering followed by Stacking using Random Forest algorithm. SEDMDroid is a Stacking Ensemble method that relies on Multi-Layer Perceptrons as base learners and four types of features: Permissions, permission-rate, monitoring system events sensitive APIs, and data flow information  (Zhu et al. 2020 ). Stacking has also been used with neural networks base learners and Dalvik instructions to predict malware  (Zhang et al.
2015). Support Vector Machine algorithm has been used to assemble the prediction output of Naive Bayes classifiers  (Palumbo et al. 2017). Mlifdect  (Wang et al. 2017 ) is an Ensemble Learning approach that predicts an app as malware if the sum of probabilities of its base learners is above a predefined threshold. Assembling the prediction of the base learners has also been investigated using Average, Maximum, Product of probabilities, and Majority Vote with features that include permissions, Standard OS and Android commands, and API-related features  (Yerima et al. 2014). Majority Voting has been leveraged to assemble classifiers trained with permission features  (Christianah et al. 2020) , and with the combination of permissions and source code features  (Milosevic et al. 2017). Soft voting has been used to combine the output of a Decision Tree, a Deep Neural Network, and an LSTM classifier that are trained using API calls, API frequency, and API sequence features.
Conclusion The literature of Android malware detection is prolific. Nevertheless, the expectation gap between the promising research results and the severe spread of malware suggests that our community needs to revisit the evaluation of the promise of state-of-the-art approaches. In this work, we contribute with a large-scale evaluation of four state-of-the-art malware detectors published at major venues, using two datasets of over 197k and 265k apps. We confirm previous results in the literature, which found that the performance of malware detectors is highly dependent on the dataset used in the evaluation. Particularly, no approach has reported the best detection results on all the settings, which casts doubts on the usability and the validity of the studied approaches in real-world settings.
Acknowledgements This work was partially supported (a) by the  Fonds National de la Recherche (FNR) ,  Luxembourg , under project  CHARACTERIZE C17/IS/11693861 , (b) by the  University of Luxembourg  under the  HitDroid grant , (c) by the  SPARTA  project, which has received funding from the  European Union 's  Horizon 2020 research and innovation program  under grant agreement No  830892 , and (d) by the  Luxembourg Ministry of Foreign  and European Affairs through their Digital4Development ( D4D ) portfolio under project  LuxWAyS.