dataesr/scientific-paragraphs-categorization huggingface dataset:
>>> autowig. controller['clanglite'] = clanglite_controller >>> autowig. controller. plugin = 'clanglite' After the generation and compilation of wrappers (using the same procedure as the one described in Section 6. 1), it enabled us to propose a new parser implementation called clanglite. This has been done by writing Python code responsible for the traversal of the AST and the completion of an existing ASG. Contrarily to the libclang parser the AST traversed by the clanglite parser contains template classes and their specializations. This parser is therefore more efficient and is selected by default in AutoWIG, as soon as the ClangLite bindings are installed.
>>> v (-1, 0, 1) Moreover, the stl/ init. py module imports all Python objects of the stl/ stl. so library at its root to simplify class names (e. g. , stl. VectorInt instead of stl. stl. VectorInt). Some additional features are automatically added in AutoWIG wrappers. For example, for functions returning non-constant references (e. g. , int& operator[] (size type pos); of the std::vector< int > instantiation), an additional wrapping is done using the following decorator. This decorator is then dealt as an overloaded method in wrappers. In this particular example, it enables to define getitem and setitem methods in the stl/vector. py module. >>> v[0] -1 >>> v[0] = -2 >>> v[0] -2 If this decorator is not written, there is no way to use the setitem method in Python. Moreover, since Python users are more familiar with Python containers, each method taking a C++ container constant reference or copy as parameter try to convert automatically Python objects into the corresponding C++ container.
Wrapping dependent libraries StructureAnalysis is a set of libraries including statistical models for the analysis of structured data (mainly sequences and tree-structured data): • StatTool is a library containing classes for the parametric modeling of univariate and multivariate data (see Figure  3 ). • SequenceAnalysis is a library containing statistical functions and classes for markovian models (e. g. , hidden variable-order Markov and hidden semi-Markov models) and multiple change-point models for sequences (see Figure  4 ). The SequenceAnalysis library depends on the StatTool library. These libraries have been extensively used for the identification and characterization of developmental patterns in plants from the tissular to the whole plant scale. Previously interfaced with AML (a home-made, domain-specific programming language), some work has been done to switch to Python.
Nevertheless, the complexity of writing wrappers with Boost. Python limited the number of available components in Python in comparison to AML. One advantage of having a statistical library written in C++ available in Python is that developers can benefit from all other Python packages. As illustrated in Figures  3 4 , this is particularly useful for providing visualizations for statistical model assessment using -for example -the Matplotlib  [22]  Python package. The StatTool library. In order to wrap a C++ library, that will be used as a dependency by other libraries, the user needs to save the ASG resulting from the wrapping process. In the StatTool case, we first generate the wrappers (using the same procedure as the one described in Section 6. 1). Then, we use the pickle Python package for serializing the StatTool ASG in the 'ASG.
>>> import pickle >>> with open('. /stat_tool/ASG. pkl', 'r') as f:. asg. merge(pickle. load(f)) After the generation and compilation of wrappers (using the same procedure as the one described in Section 6. 1), the user can hereafter use multiple change-point models [see 23, 24, for applications of multiple change-point models] in the Python interpreter. Multiple change-point models are used to delimit segments within sequences, for which the characteristics of variables (or vectors in the multivariate case) are homogeneous within each segment while differing markedly from one segment to another (e. g. , piecewise constant mean and variance for a Gaussian change in the mean and variance model). For instance, we considered the classic example of well-log data  [25, 26, 27]. >>> from structure_analysis import sequence_analysis >>> seq = sequence_analysis.
Python) have been proposed to simplify and ease the process of wrapping large C++ libraries. SWIG  [8, 28]  implements its own compiler that simplifies the process of wrapping large C and C++ libraries into a large number of different languages, and in particular R and Python. While SWIG is capable of wrapping most of the C++ features, it requires configuration files to include interface and conversion specifications. If there is a change in the library, these configuration files may become out of date. Cython  [6]  is another semi-automatic solution. Cython both enables Python users to compile Python code to C for optimizing execution of scientific code, and makes it possible for developers to call C or C++ code from Python. Cython is intensively used by several Python scientific libraries  [29, 30]  that optimized critical part of their code by writing subparts of the package in Cython.
• Strategies to convert this abstract view of the C++ code into manual or semi-automatic wrapper tools. • The generation of the Python or R bindings based on these information. The first difficulty is to parse large C++ code, and provide information on its structure. For this, tools like Doxygen or GCC-XML have been used. While Doxygen was first developed to automatically extract and render documentation of C++ libraries, it provides an XML representation of the C++ interface that can be used to describe functions and classes. Later, GCC-XML has been developed to offer a representation of a C++ library in XML using the GCC compiler. This tool has been developed for one of the first automatic library, CABLE, used to wrap the large visualization library VTK  [31]. However, maintaining such a tool is complex and GCC-XML does not support C++ 11 standard.
In AutoWIG, we use the LLVM/Clang technologies  [17]  to have the latest version of the compiler. Clang provides a full representation of the compiled library. Among the automatic tools, CABLE and WrapITK  [32]  generate SWIG configuration files to build the wrappers, Py++  [11]  generates Boost. Python code, and XDress  [12]  generates Cython files. Some domain specific tools, like Shiboken, have also been developed to wrap their large C++ libraries (in this case the entire QT libraries). While these tools provide an excellent solution for very complex libraries, they have some limitations. Some libraries rely on GCC-XML that does not support modern C++ standard. However, a new tool CastXML is currently in development. The main tools depends on configuration files and are called as executable like XDress and WrapITK.
Extensibility As stated above, the plugin architecture of AutoWIG enables nonintrusive extensibility. This is of great interest when considering the addition of other source or target languages. The addition of a target language principally consists in writing Mako templates  [19]. As an example, let consider the R language. In order to be able to propose automatic R bindings for C++ libraries, the templates written could be based on the Rcpp  [9]  library. This is particularly interesting since Rcpp wrappers are quite similar to Boost. Python ones. As a matter of fact, the implementation of a r cpp generator is of highest priority regarding future work. The major difficulty encountered is the lack of some features in Rcpp (e. g. , enumeration wrapping) and particular organization of R packages that must be taken into account.
The addition of a source language is more problematic since it could lead to addition of new proxy classes in the abstract semantic graph. For example, if the addition of the C, Objective C or Objective C++ languages should be relatively easy since it can be done using the Clang parser and C++ proxy classes, the addition of the Fortran language requires more work. In fact, for this purpose the Open Fortran Parser  [33]  could be used but it would require to reimplement the transformation of an abstract syntax tree to an abstract semantic graph. Moreover, any addition of a source language must be followed with the addition of target language generator since wrapper technologies are dependent of source languages. For Fortran, once the parser is implemented, this would require in addition to implement a generator potentially based on the F2Py [10] tool.
Toward a reference guide generator In its current stage, AutoWIG translate the Doxygen  [20]  documentation into a Sphinx one  [21]  but only incorporates it in the wrappers. This means that both Doxygen and Sphinx tools must be used to respectively generates C++ and Python reference guides. Writing a generator that would generate Sphinx compatible files containing the C++ reference guide could be of great interest since it would allow to aggregate both C++ and Python documentation within the same tool.
Installation and usage The installation of AutoWIG has been tested on Linux, MacOs X and Windows with Python 2. 7. Nevertheless, the most effective wrapping process relies on the ClangLite extension, that has not yet been released for Windows. Note that wrappers generated with AutoWIG do not depend on AutoWIG and can be built on any operating system without regarding the operating system from which the wrappers were generated. On each of these operating system, AutoWIG binaries are available using the Conda package management system. Note that these binaries require to be installed in a specific environment that will be used for wrapper generation but not for compiling these wrappers since conflicts can occur between AutoWIG's requirements and those of the wrapped library. Moreover, Docker images  [34]  can be downloaded (https://hub.
Concluding remarks AutoWIG greatly simplifies the process of incorporation of compiled libraries within scripting language interpreter. It provides the concept of ASG as C++ code abstraction data model. AutoWIG can therefore be used for C++ code introspection in a Python interpreter to discover and analyze C++ library components. This enabled us to propose an automatic generation of Python bindings for C++ libraries respecting some guidelines. This generation of Python bindings is also combined with the automatic generation of pythonic interface (e. g. , use of special functions, error translation, memory management and Sphinx formatted documentation) using Mako, a template language classically used in web frameworks. Some compilation problems led us to also to consider a tool for parsing compiler errors that is particularly useful when considering the wrapping of class template specializations.
Note that a particular attention has been payed for AutoWIG architecture: • It has been designed as a library. This choice has been made since it enables interactive wrapping of compiled libraries in the high-level scripting language, Python. This interactivity use of AutoWIG increases the user ability to supervise or debug the wrapping process and reduces the level of expertise required to use this software. • It has been designed as a plugin-oriented architecture. This choice has been made for extensibility purpose to enhance the adoption of Au-toWIG by developers by simplifying the integration process of external contribution. While only C++ to Python bindings have been implemented, AutoWIG plugin architecture eases the process of source (such as C ) or target (such as R) language addition.
Supplementary materials AutoWIG The source code of AutoWIG is available on GitHub (http: //github. com/StatisKit/AutoWIG). The documentation is hosted on Read The Docs (http://autowig. readthedocs. io). It is distributed under the CeCILL license (http://www. cecill. info/licences/Licence_CeCILL_V2. 1-en. html). CeCILL license is GPL compatible. Examples presented in this article and the documentation are reproducible available on Docker images (https://hub. docker. com/r/statiskit/autowig/tags/).
INTRODUCTION Google, with its Android platform, has been solidly integrated into the smartphone market in recent years. One of the reasons for its success is its application catalog also called "Playstore"  (Statista (2017) ) which contains, based on our estimation, more than 2 million applications. It is also the developers' favorite platform because publishing applications is both quick and free. However, the number of users of this platform make it an attractive target for spreading malware (malicious applications). Despite the protections put in place by Google, it is certain that malicious of applications exist on the Playstore such as Viking Horde (A. Polkovnichenko (2016) ), DressCode  (TrendMi-croInc (2016) ) or FalseGuide (O. Koriat (2017) ) attacks have demonstrated. This article will present our study which aims at determining the proportion of malware on Google Play.
We consider as malware any application intentionally causing harm or subverting the system's intended function (this includes Adware), as well as manipulating information without user's express consent. To perform this study, we have designed a crawler able to simulate the download of an application and to bypass the downloading restrictions of Google Play. Then, with data from static analysis, we trai-ned an Artificial Neural Network to discover new malware on the PlayStore. This neural network bases its detection on features that are extracted by reversing these applications. In the following paper, we will detail the different techniques involved in performing our study. It is organized as follow : Section II, the crawler ; Section III, the features used for static analysis ; Section IV, the feed and forward neural network ; Section V the results.
PLAYSTORE CRAWLER As we are conducting a study on Googles Playstore, we first need to retrieve a huge set of applications from it (large enough to be statistically representative). To do so, we have developed a Python script to download applications massively directly from the Playstore. We describe here the restrictions implemented by Google and how we bypassed them. First of all, there is no exhaustive list of the applications available on the Playstore. Moreover, Google tends to bring out the most popular applications and to hide those less successful. Secondly, even though the communication with Playstore servers go through HTTPS, to only allow Android devices with a valid Google account, a proprietary protocol is used on top of it. This protocol is based on Google's protobuf  (Google (2017) ), and has not been officially described as of now, but has been, for the most part, reverse engineered (Figure  1 ).
The first one is expensive and the second one does not respond to our problematic, crawling the whole PlayStore. On GitHub, we can find a Python script (dflower (2014)) that is able to communicate with the Playstore. This script implements the basic functionalities of the store (search for applications, get the details, download,. ) and needs only two elements to work with: a valid Google account and an Android Device ID (a unique token generated for each Android device)  (Android (2017c) ). However, the tool is not designed for mass downloading and we modified it to suit our needs. To avoid being banned because of non-human behaviours, we crowd-sourced the creation of 30 activated Google accounts (an Android device is required to validate an account before it is allowed to download from the Playstore, and you can only create a few accounts from one device) and we try to put as much time as possible between two downloads from the same account (multiple minutes).
To speed up the whole process, we split the dictionary between multiple processes and each of these spawns a thread provided with an account/device ID pair for each app to download (see Figure  2 ). The crawler is designed to be scalable. Provided that we have enough accounts and Device IDs, it is possible to distribute the crawling across multiple machines. In our case, to synchronize the different pro-cesses, we store the applications inside a Cassandra NoSQL database  (Apache (2008) ) and before downloading a new one, we check if the application is absent from the database. The crawler has been used for two purposes. First, in October 2016, it was configured not to download applications, but just to find them. It has generated a list of more than 2 millions applications. In January 2017, we used it to download and extract the features (see section 3) of 3,650 applications, which constitute our sample for this study.
Extraction We perform static analysis, and therefore, we extract all the features that can be interesting for classification directly from the APK file. To reverse applications, we use the set of tools Androguard  (Bachmann (2015) ) written in Python. We provide the type of features we extract and why. Our static analysis follows a similar pattern to what was presented by P. Irolla and E. Filiol at Black Hat Asia Conference (2015)  (Irolla and Filiol (2015) ). • General Information: This information is not directly used to perform the analysis, but more as a way to identify and contextualize it. We extract these information mainly from the application manifest  (Android (2017a) ). First, we retrieve the application common name (ex: Facebook Messenger) the package name (com. orca. facebook), and the version number.
Both of these are supposed to identify the application, but considering it is easy to repackage an Android application, we use the SHA-256 hash of the APK instead. We also extract the certificate used to sign the application (to be published on the Playstore, an application must be signed). From this certificate, we can extract interesting information on the developer. The other things we extract are the information relative to the SDK (Android (  2012 )) (minimal/maximal/target version), the intent for the statically defined receivers (communication between applications), intents for the activities (starting a foreground process inside an application) and intent for the services (background processes). The last thing retrieved is all the URLs statically defined in the application. • Classification Information: These are the features on which the detection is based.
They are retrieved by decompiling the. dex files in the application. These files store the java bytecode that will be executed by dalvik (or Android RunTime), the android Java virtual machine  (Android (2017b) ). We chose these features because, based on other researcher's work, they seemed promising for static analysis (see: G. Canfora and Visaggio (2015b); G. Canfora and Visaggio (2015a); D. Arp and Rieck (2014) ). Most of the information we extract is based on the opcodes (the instructions executed by dalvik) themselves, without the operands. The first thing we use is opcodes frequencies (G. Canfora and Visaggio (2015b) ). As a more representative value, we store trigrams (An N-gram is a sequence of N adjacent opcodes) frequencies (G. Canfora and Visaggio (2015a) ). In Figure  3 , the opcodes are (invoke-static, move-result-object, if-eqz, invoke-direct, returnobject, const/4, goto), and the corresponding trigrams would be ([invoke-static:move-result- object:if-eqz], [move-result-object:if-eqz:invoke- direct], [if-eqz:invoke-direct:return-object], [invoke-direct:return-object:const/4], [returnobject:const/4:goto]) To use several Android API methods, Android applications have to ask for certain permissions (use network, bluetooth, access user information,.
). Malware also have to ask for these permissions, and some permissions give access to more potentially malicious behaviour, and are therefore a good way to analyze applications (D. Arp and Rieck (2014) ). Finally, we extract the Android API call sequence. The API is a software interface (a set of functions) used by developers to perform device related tasks (sending SMS, manipulating the UI, etc. ). Some of these tasks can be maliciously employed. These calls were found to also be pertinent to detect android malware (D. Arp and Rieck (2014) ). For Figure  3 , API call would be api function call1, api function call2, as well as the constructor of api type1 From a collection of applications, we extracted these features. There is a total of 228 dalvik opcodes. The applications were constituted on average of 80,000 to 100,000 opcodes, but a few reached more than a million of them.
TF is the feature presence frequency in our malware database and the IDF is the feature presence frequency in the benign dataset. We have then an indicator that grows if the feature is well represented inside the malware database and diminish if the feature is well represented in the benign application database. t f (t, d) = f t,d ∑ t ∈d ( f t ,d ) (1) id f (t, d) = N |{d ∈ D : t ∈ d}| (2) t f id f (t, D) = t f (t, d) * id f (t, D) (3) After calculating our modified version of the TF-IDF on each feature and sorting the results from the most pertinent to the least pertinent, we plot the TF-IDF histogram for each section (Figures  4, 5  and 6 ). We can select the most important features given on the results distribution and, by cutting a section for each graph, we then obtain 12175 features which will be our entries.
CLASSIFICATION 4. 1 Training Dataset We use an artificial neural network to classify applications. To build it, we need a set of applications that are already classified. We use the Drebin Dataset (D. Arp and Rieck (2014) ). The Drebin Dataset is widely used by the Android malware researcher community. At first we used the 5585 malware and 2201 benign applications as our training dataset. We have conducted a study on this initial set  (Irolla and Dey (2017) ) that show that 42. 5% of these applications can be regrouped into 592 (584 malware, 8 benign) sets of repackaged applications with the same code and only resources and a few strings that differ from one another. Therefore, we removed the duplicates from the Drebin Dataset, and we end up training on a set constituted of 2891 malware APK and 2178 benign ones.
Testing Phase We trained our neural network to maximize our generalization capabilities. The optimal training was found thanks to the cross validation method. Our criteria for this training is the MSE (Mean Squared Error)  (Lehmann and Casella (1998) ). We select the training stage where the MSE on a foreign dataset is at its lowest (See Figure  8 ). MSE = 1 n n ∑ i=1 ( e i -e i ) 2 (4) The best neural network among all the created neural networks is the one with the best accuracy. The accuracy criteria is: Acc = T P + T N T P + T N + FP + FN (5) With TP, number of True Positive, TN, True Negative, FP, False Positive and FN, False Negative. For a health state, instead of a ROC analysis, we privileged accuracy with an ANN giving a 98. 03% accuracy with equal false positive and false negative rates.
Some testings showed that using a C++ program greatly reduces the extraction time for the opcodes (this program is still work in progress). Additionally, once it will be easily deployable and fully documented we plan on releasing the crawler to the OpenSource community (https://github. com/AlexandreDey/cygea-playstorecrawler). Considering the classifier, it shows great accuracy for only light computation time. The TF-IDF variant employed here also allowed us to target more precisely the important features without the need of deep knowledge in the Android malware domain. However, moving to a deep learning algorithm might be beneficial for the accuracy rate. Classification capabilities could be enhanced by using a larger training dataset constituted of more actual and complex malware. Finally, we are sure that Google's Playstore houses a certain number of malware, and we estimate that this number is between 40,000 and 60,000 applications.