dataesr/scientific-paragraphs-categorization huggingface dataset:
We discuss the irregularities of datasets used in experiments, questioning the validity of the performances reported in the literature. This article focuses on providing guidelines for designing debiased datasets. First, we propose guidelines for building representative datasets from unlabeled ones. Second, we propose and experiment a debiasing algorithm that, given a biased labeled dataset and a target representative dataset, builds a representative and labeled dataset. Finally, from the previous debiased datasets, we produce datasets for experiments on Android malware detection or classification with machine learning algorithms. Experiments show that debiased datasets perform better when classifying with machine learning algorithms. Index Terms-Datasets, malware, experiments.
I. INTRODUCTION D UE to the growing number of devices equipped with the Android operating system, attackers have developed a large number of malicious applications that may steal personal information, demand ransoms, or make smartphones unavailable. In response to that phenomenon, researchers have tackled the question of determining whether an application is malicious or benign and the question of classifying an application inside malware families. Most of the work in this area follows an experimental approach, and the proposed solutions must be evaluated on datasets mirroring the possible set of Android applications. Over time, some datasets become extensively used and at some point, become outdated even if they are the lat-est to provide good information  [1] -  [3]. Results are often impressive, with detection ratios greater than 99% for some articles  [4] -  [6].
However, for these results to have any practical significance, the evaluation methodology and the nature of the datasets should be thoroughly inspected. In particular, special care should be taken to ensure that the experimental datasets are representative of the overall population of Android applications in the wild. Pendlebury et al. [7]  identified the problem of bias in test datasets used to evaluate machine learning classifiers, which leads to overestimating the performance of an algorithm compared to its use in real-life scenarios. To evaluate an algorithm's performance on a population of applications, we need a dataset that is representative of the population and that is labeled with the sought answers. Such answers require manual analysis of each application which is often a non-trivial task.
This article presents four main contributions: • A methodology for building, by random sampling, a nonlabeled dataset that is representative of the considered population of Android applications; • A debiasing algorithm that, given an initial labeled and non-representative dataset and a representative target dataset, outputs a labeled and representative dataset; • An experimental study on different datasets that have been debiased. • An evaluation of the usefulness of debiased datasets when classifying malware/goodware samples using machine learning algorithms. The article is structured as follows. Section II provides background information on the various datasets available to the research community and the type of experiments performed with them. Section III describes related work on the mitigation of irregularities that affect datasets.
A. Sources, Archives and Specific Datasets Websites called "markets" are the different Android application distribution platforms available to the public, e. g. , Google Play, Aptoide, Cafe Bazaar, F-Droid, MiKandi, Appland. Google Play, the first and largest of them, is supposed to be a source of new benign applications (goodware). Indeed, Google uses services such as the bouncer 1 and Google Play Protect, 2 which analyze applications before publishing them. There are also country-specific markets, particularly several Chinese markets (e. g. , Baidu, Tencent, 360, 25PP, HiAPK, Wandoujia), and vendor-specific markets (e. g. , Huawei, OPPO, Xiaomi, Lenovo). These markets evolve through time, as new applications get published and old ones are withdrawn. There are also archives datasets where applications are never removed.
AndroZoo  [8] , APKMirror 3 and APKPure, 4  are examples of archives. Lastly, specific datasets provide a variety of applications that share a common characteristic, for example, being a malware, being repackaged applications, or being a specific type of malware (ransomware, dropper, etc. ). Genome  [2] , Drebin  [1] , Andrubis  [9] , Kharon  [10] , AMD  [3] , CICAndMal2017  [11] , VirusShare  [12] , UpDroid  [13] , Koodous  [14] , and VirusTotal  [15]  are examples of specific datasets (and mainly malware datasets). Some datasets are mostly static, meaning they are not regularly updated and therefore become rapidly obsolete.
B. Experimental Scenarios We have identified three types of scenarios typically experienced by security researchers. Each of these scenarios has different issues and therefore requires the use of datasets with different expected properties. 1) Markets Profiling: Researchers may want to estimate a statistical characteristic of a market  [8] ,  [16] ,  [17] , e. g. , "What proportion of Google Play applications have an API version lower than some threshold?. " For this case, one needs to obtain a dataset which is representative of the market, where results of analysis on this dataset can be extrapolated to the whole market. Section V gives more details on how such a representative dataset should be extracted. 2) Algorithm Evaluation: Some experiments aim to compute a characteristic of interest from an application.
Examples of interesting characteristics include detecting malicious behavior in an application  [18] , whether the sample is repackaged  [19] , identifying its malware family  [20] , detecting similar applications  [21] , whether it leaks personal information to the network  [22]. To answer these questions, researchers implement algorithms using various analysis techniques, and run them on a dataset to estimate their precision. This dataset should be representative and labeled, i. e. , the characteristic of interest for each application should be known beforehand. For any of the three kinds of experiments above, building an appropriate dataset is a non-trivial task. In the rest of the paper we take as hypothesis that a researcher needs a dataset for evaluating a detection algorithm.
A. Class Imbalance Class imbalance occurs when a dataset contains significantly more units of one class than of others  [23]. An overrepresented class may push a learning algorithm towards preferring this class. For malware detection, if there are more samples of goodware, this has the effect of decreasing the recall but improving the precision  [7]. This problem has been addressed in the machine learning community  [24] : the overrepresented class is undersampled, i. e. , some elements are removed while the underrepresented class is oversampled. The undersampling can be random or rely on a more sophisticated method, such as removing elements in dense clusters  [26]. The most popular oversampling technique is SMOTE  [27]  that creates new instances from the interpolation of close samples. This technique cannot be used for malware datasets, as one cannot forge a new malware sample with specific characteristics automatically.
B. Presence of Clones Another irregularity is the presence of clones, where the abundance of several very similar apps in the training set prevents the algorithm from learning unique units  [25]. Because experiments do not usually share the list of used applications, it is not possible to check the presence of clones in their datasets. Pan et al. [28]  show a method to reduce redundancy in a dataset by extracting representatives of cloned units using mutual information and relative entropy.
In  [33] , Zhang et al. observes that malware samples often implement the same functionalities over time but change their implementation because of the evolution of APIs or changes of third-party libraries. These changes allow such malware to avoid detection by older classifiers. One way to limit this problem is to choose more semantic than syntactic features, as stated by Zhang et al. in  [33]. Finally, in  [34] ,  Xu et al. propose to enhance malware detection over time by making necessary updates to their detection models with evolving feature sets. For that purpose they maintain a pool of different detection models initialized with a set of labeled applications using various online learning algorithms. The previously cited works cannot solve the problem of researchers having to evaluate their algorithm on an up-todate dataset.
Indeed, this problem is different than adapting a classifier to an evolving market -even if this problem is of independent interest. The researcher would primarily focus on getting a dataset that mimics the whole set of applications that he targets. Additionally, the dataset should be labeled, for example with a malware/goodware label if the study concerns a detection algorithm. These labels can be missing in the source from which applications are picked. As a consequence, we propose in the following to define the notion of representativity of a dataset against the population it represents. A representative dataset has to be used to validate an experimental results at the time of the experiment. The representativity property is linked to a population at a period of time. A dataset that is representative at a date will not necessarily be so in the future because the population may change significantly.
IV. REPRESENTATIVITY In this section, we describe the notion of representativity of a population that we will be aiming for in the rest of this article. The notion of population refers to a set of applications that a user would want to consider for his use case. As an example, a user who wants to detect malware in the Google Play store would want to take the Google Play store as population. Nevertheless, it is very difficult to get a snapshot of such a market. Other markets of applications could be considered  [17]  but the same problem remains. As a consequence we chose in this paper AndroZoo as population. It is not an hypothesis that reduces the contributions of this paper: the proposed techniques work more generally beyond a particular reference dataset. From a general perspective, we consider a population P of N applications.
Intuitively, a dataset D of n elements is representative of P if n is significantly smaller than N and if D looks like P for the following purposes: 1) evaluating the proportion p of applications exhibiting a specific characteristic on D should yield similar results as evaluating the same proportion p on P; 2) evaluating an algorithm on D should yield similar results as evaluating the same algorithm on P. Intuitively, we could consider that a dataset D is representative of P according to a single characteristic if the proportion of this characteristic is equal in P and D. However, the exact equality of the proportions in P and D (i. e. , p = p) cannot always be achieved in practice. For example, if p = 1/N (i. e. , only one element has this characteristic in P), then the only way to have exactly the same proportion in D is by selecting all N elements from P.
This is in contradiction with the first desired property of a representative dataset. Hence a margin of error δ must be taken into account, which is the maximum difference between p and p that we allow considering that D is representative of P. Intuitively, a dataset is representative of a population if it has the same proportions, up to a maximum difference of δ. This notion can be easily extended to multiple characteristics. Consider a set of d characteristics {c 1 ,. , c d } that can take respectively v 1 ,. , v d values. This generates v 1 ×. ×v d different combinations. In the following, we call these combinations classes, and we denote K this set of classes. Intuitively again, we could consider that a dataset D is representative of P according to a set of classes K if the proportion of each class k ∈ K is equal in P and D.
V. SAMPLING REPRESENTATIVE DATASETS In this article, we consider the population of Android applications to be AndroZoo, because this is one of the largest datasets available, it is regularly updated from other markets and malware databases, and no application is removed from it. At the end of 2020, AndroZoo contained 13 696 936 different APK. In this section, we focus on the construction of a nonlabeled δ-representative dataset of AndroZoo by sampling, i. e. , random drawing. The standard way to construct a representative dataset consists of drawing n random samples without replacement  [36]. Because elements are drawn randomly, a very large fraction of the population would be necessary to ensure, with 100% confidence, an error smaller than δ. In statistics, one is generally more interested in probabilistic guarantees such as "with 95% chance, the error is smaller than δ.
A. Dataset Size for a Single Boolean Characteristic Once the expected margin of error δ and the expected confidence level C are defined (according to the requirements of the experiment), a lower bound on the size n of the dataset D can be calculated by using Equation (  1 ). This equation is obtained by rearranging the margin of the classical error formula  [37]  using the finite population correction  [36] , where z(C) is the standard deviation value at C confidence level for the standard normal distribution with mean 0 and standard deviation of 1 (the derivation of this equation is detailed in the appendix). n ≥ N 4(N -1)δ 2 z(C) 2 + 1 -1 (1) This computation is based on some light assumptions, notably that n is not too small (at least 30  [38] ) and that the proportion p is not very close to 0 or 1.
In such cases, other confidence interval formulas can be used, like the Hoeffding inequality  [39]. The effect of N is limited when it is large. The minimum size of a random sample for 3 014 560 applications in Google Play with the values δ = 0. 01 and C = 99% is n = 16 497 while it is n = 16 568 for AndroZoo, a dataset four times larger. For arbitrarily large datasets, n = 16 588 examples are sufficient. The influence of N is more significant for smaller populations. For example, with a population of N = 15 000 elements, one only needs n = 7163 examples.
B. Dataset Size for one or Multiple Characteristics The calculation of the size of the dataset given in the previous subsection only holds for a single boolean characteristic, so when there are only two classes. If one is interested in several characteristics or in a non-boolean characteristic, then there are more classes for the dataset to be representative of, and intuitively it means that the minimal size of a dataset representative of a population is larger. As with a single boolean characteristics, the margin of error can be used to get a sample with an error up to δ. The Bonferroni correction  [40]  states that correctly representing a set of classes K with a confidence level C using random sampling is more likely than correctly representing one boolean characteristic with a confidence level 1 -(1 -C)/(|K| -1).
Therefore, we can modify Equation (1) by replacing z(C) by z(1 -1-C |K|-1 ). Hence, the dataset D will correctly represent |K| classes with a margin error δ with a probability higher than C if n satisfies the following inequation: n ≥ N ⎛ ⎜ ⎝ 4(N -1)δ 2 z 1 -1-C |K|-1 2 + 1 ⎞ ⎟ ⎠ -1 (2) Remark that, in the case of a single boolean characteristic, |K| = 2 and equations (  1 ) and (  2 ) are equal. In the experiments of Section VI, we will observe 2411 non-empty classes in AZ20 30k , the reference dataset. With the values δ = 0. 015, C = 99% and |K| = 36 864, the minimum size of a random sample is n = 29 099 for Google Play, n = 29 320 for AndroZoo and n = 29 383 for datasets with arbitrarily large N.
VI. EVALUATION OF EXISTING DATASETS We now assume that an algorithm or a machine learning model evaluation has to be performed, as described in Section II-B. Such evaluation needs a labeled dataset, i. e. , one whose ground truth is known. It should also be representative of the entire population. Otherwise, results could be erroneous due to biases in the dataset. In the case of Android applications, some characteristics like "being malware" are not easily computable, and only small datasets or old datasets contain such information. For a given hard-to-compute characteristic, we separate the infinite number of other characteristics into two groups. In the first group, the characteristics that are likely to be related to the hard-to-compute characteristic. For example, using a cryptographic library could be related to being a ransomware or not.
In the second group, the characteristics are not related to this hard-to-compute characteristic. We would like the datasets used to be homogeneous to the population for the characteristics of this second group. We now propose a homogeneity test to assess if a dataset is homogeneous to a population for a finite set of characteristics. To determine this finite set, we consider the hard-to-compute characteristic "being a malware" and we choose a set of characteristics that are not related to it. Then, we test the homogeneity of existing datasets with the population for this set.
A. Choosing Characteristics The more characteristics are selected, the more combinations there are. Since this number of combinations is exponential in the number of characteristics, it can quickly grow bigger than any dataset size as the number of selected characteristics increases. Therefore, it is necessary to limit the number of characteristics. Besides, numerical characteristics should be reduced to a few values to reduce this combination's number. For example, the APK size can be projected to a few intervals of size to distinguish small, medium, and large applications. As a consequence, in the following experiments, we choose to rely on the following set of characteristics that are not related to security: • Specifies User-Agent (bool) These characteristics can be easily extracted using Droidlysis  [41] , 5 a property extraction tool.
With such characteristics, 36 864 (4 × 9 × 2 10 ) classes would be possible, but in practice, we observe 2874 non-empty classes in the three studied datasets (AZ20 30k , D mix and AMD) presented later in Figure  1. • APK size (4 values) 0) 1 MB < size 1) 1 MB ≤ size ≤ 5 MB 2) 5 MB ≤ size ≤ 20 MB 3) size > 20 These characteristics are diversified and easy to evaluate statically, limiting the computational cost of our experiment. It would be possible to add some dynamic characteristics, such as the number of executed activities or methods, but it would require executing each application. These characteristics will be used to study the homogeneity of the datasets of the literature against an extract of the population, AndroZoo in 2020.
B. Statistical Test of Homogeneity for Datasets To evaluate whether a dataset D can be considered to be not statistically homogeneous to a population P for a finite number of characteristics, we propose to use the χ 2 statistical test of homogeneity  [42]  and compare D with a second dataset D r , a large dataset randomly drawn from P (see Section V). The null hypothesis of the test is H 0 : D and D r are drawn from the same population P (with replacement). As other statistical tests, the χ 2 test can only reject this null hypothesis but not confirm it. If the null hypothesis is rejected, we will admit that the dataset D is not homogeneous to D r , and therefore dissimilar from the population P for the characteristics under consideration. To apply this test, we consider a finite set of classes K.
χ 2 = k∈K m D,k -E D,k,D r 2 E D,k,D r + m D r ,k -E D r ,k,D 2 E D r ,k,D A low value for a χ 2 test means the two sets of data D and D r have very similar class proportions. On the contrary, a large 5 https://github. com/cryptax/droidlysis value (larger than a critical value that depends on the number of classes and the confidence level) indicates a statistically significant difference. A χ 2 test is associated with a p-value. A low p-value (usually under 0. 05) indicates that one can reject the null hypothesis with a certain confidence. A high p-value, however, does not confirm the null hypothesis, but rather produces an inconclusive result. In the rest of this section, we investigate whether labeled datasets commonly used in the literature can be considered homogeneous to all Android applications according to the χ 2 test and whether they can be considered δ-representative for some low δ.
C. Studied Android Datasets In this section, we focus on the following datasets: • 6 malware datasets: Drebin  [1] , AMD  [3] , as well as VS 2015, 2016, 2017 and 2018, containing all the malware collected by VirusShare for each of these years; • 2 groups of datasets extracted from AndroZoo  [8] : -AZ19 100k , 100 000 applications randomly drawn in 2019, and the subsets AZ19 100k 2015, 2016, 2017, 2018 restricted to applications from each year; -AZ20 10k , AZ20 20k , and AZ20 30k three datasets randomly drawn from AndroZoo in 2020, containing respectively 10 000, 20 000 and 30 000 applications. • 1 mix dataset: D mix composed of 5560 malware from Drebin and 11 120 goodware from 2018 extracted from AndroZoo. D mix mimics the datasets used in previous machine learning papers  [43] ,  [44]. In this article, we use AndroZoo as the population, because it is the easiest source to draw a large number of applications.
Our goal is to determine which datasets commonly used in the literature can be considered as representative of the population. AndroZoo provides 13 696 936 different APKs applications. Due to its size, it is not reasonable to calculate exactly the distribution of AndroZoo. Thus, we represent here AndroZoo through a representative dataset AZ20 30k of 30 000 randomly chosen from it (corresponding to δ = 0. 015 and C = 99% for all 36 864 theoretical classes, cf. Section V-B). We performed the comparison of AZ20 30k with the datasets of the literature, presented in Section II. As an illustration, Figure  1  shows the proportion of applications in each of the 2874 non-empty classes of three datasets: the reference AZ20 30k (in blue), the mix dataset D mix (in orange) and AMD (in green). The classes are sorted in descending order of size for AZ20 30k.
This figure already suggests that the three datasets are very different. We confirm this intuition by performing a χ 2 test with the AZ20 30k dataset. Table  I  presents the dataset size, the number of classes represented only in AZ20 30k (and not in the studied dataset), the number of classes common to AZ20 30k and the studied dataset, the number of classes represented only in the studied dataset (and not in AZ20 30k ), the maximum and average δ of class proportions, the value of the χ 2 test, the associated p-value, and the conclusion (or lack of) of the test between the studied dataset and AZ20 30k. All datasets, except the last extract of AndroZoo, can be considered as not homogeneous to AZ20 30k ,  rejecting the null hypothesis with a p-value < 0. 05. We can also remark that extracts of AndroZoo have a low average δ value, smaller that 0.
VII. DATASET DEBIASING ALGORITHM The previous section strongly suggests that the datasets regularly used in Android malware scanning experiments are not representative of the entire current population of Android applications. Definitely, they are not for the common characteristics we have studied. Nevertheless, these datasets are valuable because they contain samples for which the characteristic of interest is already computed. We are now interested in producing a representative dataset of a population for a finite set of classes K from an initial dataset B. Our motivation is to not have to recompute computationally expensive characteristics on a randomly drawn dataset but to take maximum advantage of B's labels. As seen in the previous experiments, the χ 2 test can show that datasets deviate significantly from a population.