# TrojanDetector: A Multi-Layer Hybrid Approach for Trojan Detection in Android Applications
## 1 Introduction
A dire need for a multi-layer hybrid approach (combining both static and dynamic analyses) that could potentially address the limitations of these individual single-layer techniques is observed [6]. Android has its own local security mechanism based on permissions and app isolation. These two features, i.e., permission and app isolation, enforce access control to security resources and operations and keep away from an application that can interfere with the execution of another. However, these schemes are not found too secure for two reasons: they were found ineffective at warning users [7], and Trojan creators can bypass these detection methods [8]. To address these issues, methods based on machine learning have gained significant popularity in recent years. In this paper, we propose a novel approach, namely, TrojanDetector, to detect android malware or Trojans in a multi-layer (application, package, and user's layer) and hybrid way. TrojanDetector collects execution data of a set of known sample Trojan and benign Android applications to generate patterns of individual system calls, sequential system calls with different calling depths related to file and network access, and so on. TrojanDetector builds up a malicious pattern set and a normal pattern set for malware detection and benign application judgement by comparing the patterns of Trojan and benign applications. When TrojanDetector needs to detect an unknown app, it uses a dynamic method to collect its run-time system call data in terms of individual and sequential system calls with different depths. Then we extract the target patterns of the unknown app from its run-time system calling data. Using our chosen machine learning classifiers trained on both the malicious pattern set and the normal pattern set, TrojanDetector classifies the unknown app as either Trojan or benign. The proposed approach is a generic detection method suitable for several types of Android malware detection. The TrojanDetector uses SVM as the classifier and achieves the highest accuracy of 96.64%. Extensive experimentation shows the effectiveness of our proposed Trojan detection scheme. Specifically, the contribution of this paper can be summarised as below:* This work presents a multi-layer hybrid approach, i.e., TrojanDetector, to detect the Trojan's misbehaviour in Android applications from three different Android levels based on the selected features and then apply multi-classifiers to train the model. * This work uses four machine learning classifiers (random forest, support vector machine (SVM), logistic regression and, decision tree) for accurate Trojan detection in Android applications. * This work performs an extensive experimental evaluation to identify the usefulness of the proposed approach and gauge the accuracy of the machine learning approach using publicly available data sets of CCCS-CIC-AndMal-2020 [9], Contagio-Mobile [10], and Virusshare [11] of Trojan applications. The remainder of the paper is organised as follows. Section 2 briefly overviews related work. Section 3 presents the proposed approach and defines Trojan characteristics that detect known and anomalous behaviours. Section 4 presents an experimental evaluation and discussion of the proposed approach. Section 5 concludes and presents the direction for future work.