Title: NEURAL MALWARE CONTROL WITH DEEP REINFORCEMENT LEARNING

Abstract: Antimalware products are a key component in detecting malware attacks, and their engines typically execute unknown programs in a sandbox prior to running them on the native operating system. Files cannot be scanned indefinitely so the engine employs heuristics to determine when to halt execution. Previous research has investigated analyzing the sequence of system calls generated during this emulation process to predict if an unknown file is malicious, but these models require the emulation to be stopped after executing a fixed number of events from the beginning of the file. Also, these classifiers are not accurate enough to halt emulation in the middle of the file on their own. In this paper, we propose a novel algorithm which overcomes this limitation and learns the best time to halt the file's execution based on deep reinforcement learning (DRL). Because the new DRL-based system continues to emulate the unknown file until it can make a confident decision to stop, it prevents attackers from avoiding detection by initiating malicious activity after a fixed number of system calls. Results show that the proposed malware execution control model automatically halts emulation for 91.3% of the files earlier than heuristics employed by the engine. Furthermore, classifying the files at that time improves the true positive rate by 61.5%, at a false positive rate of 1%, compared to a baseline classifier.

1. INTRODUCTION:

Malicious software, or malware, is a serious threat to computer users. As a first line of defense, users and organizations rely on commercial antimalware (i.e., antivirus) products to detect malware on their computers, and the antimalware engine is a key component of these malware detection products. Prior to allowing an unknown file to be executed on the native operating system, the antimalware engine often tries to detect malware using two main approaches. First, static analysis employs malware "signatures" (i.e., rules) to scan the unknown file to search for malicious byte sequences in the file without execution. Next, the engine utilizes one form of dynamic analysis called emulation to execute the file in a lightweight sandbox. Lightweight emulation does not analyze the unknown file in a full virtual machine (VM). Instead, the emulator mimics the response of a typical operating system. If the engine can detect malicious behavior during emulation, the antimalware system blocks the file from being executed on the native operating system and alerts the user that the file they are trying to install is malicious. As a result, the user's computer is not infected.

Previous dynamic analysis research has focused on analyzing the sequence of system application programming interface (API) calls made by the unknown file during emulation. Typically, the authors propose a recurrent, deep learning model to discriminate between the behavior of malicious and benign files. Pascanu et al. (2015) used a recurrent neural network (RNN), or an echo state network (ESN), in combination with either a logistic regression classifier or a multi-layer perceptron (MLP) to detect malware. Athiwaratkun & Stokes (2017) replace the RNN with a long short-term memory (LSTM) recurrent network or a gated recurrent unit (GRU), and they also propose a character-level convolutional neural network (CNN) to predict if an unknown file is malicious. A CNN followed by an LSTM is proposed for this task by Kolosnjaji et al. (2016) . In all these solutions, the authors consider a fixed-length input buffer containing the events executed from the beginning of the file. The length of this pre-defined window, ν, varies depending on the study with ν ∈ {50, 100, 200, 65000}.

In this paper, we propose a novel algorithm, based on deep reinforcement learning (DRL), to overcome this limitation and learn the best time to halt the engine's emulation to predict whether the unknown file is malicious or benign. DRL has been used previously to create adversarial samples to attack a malware classifier (Anderson et al. (2018) ). Initially, we tried to train a classifier to halt emulation based on the previous t f (e.g., 200) behavioral events, but the accuracy of this classifier was not sufficient to effectively halt the emulation. To the best of our knowledge, this is the first paper to propose using deep reinforcement learning to protect users from malware. This DRL-based neural network, combined with an event classifier and a file classifier, learns whether to halt emulation after enough state information has been observed or to continue emulation if more events are needed to make a highly confident prediction. Unlike previously proposed solutions, the DRL algorithm allows the engine to decide when to stop on a per file basis.

Results from analyzing a collection of malware and benign files demonstrates a significant improvement in the early stopping of the execution of the file. The DRL-based system halts execution of 91.3% of the files earlier than heuristics used by the production antimalware engine. When the execution is stopped by the DRL model, the true positive detection rate exhibits a relative increase of 61.5% at a false positive rate of 1.0% compared to the best performing baseline model proposed in Athiwaratkun & Stokes (2017) . Our contributions include the following: 1) we propose a deep reinforcement learning-based system which predicts when to stop emulating an unknown file, 2) we show that the proposed system significantly outperforms several recent neural malware classification systems and 3) we provide a theorem and prove that the proposed DRL-based model outperforms these earlier baseline models for malware classification.

2. BACKGROUND:

Engine System Call Events. The original data for our research was collected by scanning a large collection of Windows Portable Executable (PE) files with the anonymized for submission production antimalware engine. The engine collects behavioral events, e t , observed during emulation. Most of the events are associated with APIs invoked during execution, but other behavioral events, such as unexpected instructions or constructs, are also captured. In our data, the engine records 114 event types, e t ∈ {0, • • • , 113}, ranging from file IO, registry APIs, networking APIs, thread or process creation and control, inter-process communication, timing, and debugging APIs. The behavioral events are logged using a special version of the antimalware engine, and the files are labeled with L ∈ {0, 1} where 1 corresponds to a malware file and 0 indicates that the file is benign. Additional details on emulation and scanning can be found in Appendix A, and the threat model is given in Appendix B.

Conventional Reinforcement Learning. Conventional Reinforcement learning is normally formulated as a stochastic Markov Decision Process (MDP). There are four main components in a standard reinforcement learning structure, including States, Actions, Rewards and Policy. Each of them play a different role in formulating the RL environment. A general interpretation is that reinforcement learning is a technique to help an agent learn what is the best action and policy to take such that its expected rewards/penalties can be maximized/minimized under a stochastic MDP environment (Sutton & Barto (1998) ). Conventional reinforcement learning definitions are provided in Appendix C.

3. NEURAL MALWARE CONTROL AND IMPROVED FILE CLASSIFICATION:

An overview of the proposed DRL-based system is shown in Figure 1 , and it has two main components: the execution control model and the improved inference model. An unknown file is emulated by the antimalware engine, and this generates a sequence of behavioral file events, E.

The execution control model processes E and is responsible for controlling the file's execution.

If the execution control model can make a confident decision that the file is either malicious or benign, the execution is halted. As it is received, each individual event e t is first processed by an event classifier which makes a prediction, y e,t , indicating whether or not the most recent event history includes malicious activity. Initially, we tried halting the emulation based solely on the event classifier's output, but the classifier's accuracy was not sufficient to accomplish this task and motivated the need for deep reinforcement learning. Even though it is a weak signal, y e,t is used to construct a reward signal for the DRL model, which then produces the execution control signal, h t , indicating if the file execution should be halted or allowed to continue. The primary purpose of the DRL model is to better control the file's execution. However, we also found that it can be used to significantly improve the overall classification of an unknown file. This improved prediction, y RL,t , which indicates whether the file is malicious or benign, is generated by the improved inference model. The improved inference model boosts the weak predictions from the event classifier, y e,t , based on h t and the output of a baseline file classifier which offers an initial estimate of the probability, y f , that the file is malicious based on the initial 200 events generated by the file.

4. DEEP EXECUTION CONTROL:

The details of the execution control model in Figure 1 are depicted in Figure 2 . The input is e t , and the outputs are h t and y e,t in both figures. As each event is received, it is inserted into the event queue, a first in, first out (FIFO) queue. The event classifier then predicts y e,t for the most recent subsequence stored in the event queue. Since the output layer of the event classifier is a sigmoid function, y e,t is the probability that the most recent subsequence of behavioral events corresponds to malicious activity.

The DRL model depends upon its states, actions, and rewards. The state, s t , includes information related to all the events received up to and including the most recent event. For each e t , the event classifier's prediction y e,t is used as part of the DRL model's reward function, r t . The actions for the DRL model include continuing and halting file execution. Based in part on y e,t and s t , the DRL model generates separate Q values which are the estimated expected discounted rewards associated with these actions. The Q value signals are noisy and cannot be used directly. The action state model filters the Q values to generate the halting signal h t , which is then used by the antimalware engine to stop the file's execution.

Event Classification. The recurrent model structure which is used for the event classifier, and later for the baseline file classifier, is shown in Figure 3 . For each new event, the event classifier makes a prediction, y e,t , that the behavior associated with the most recent t f behavioral events is malicious. This event subsequence is stored in the event queue.

Each event in the event queue is input to an embedding layer, and the result is then input to a recurrent layer. We use a recurrent neural network (RNN) for the recurrent layer. As proposed in Pascanu et al. (2015) , the recurrent layer's hidden state is input to a max-pool layer which is able to better detect malicious activity within the subsequence. We next construct a sparse binary, feature vector consisting of a bag of words (BOW) representation of the event subsequence (114), the final hidden state of the recurrent layer which is the recurrent layer's embedding (1500), and the output of the max-pool layer which is the max-pool embedding (1500). This feature vector is then input to a shallow neural network in the classifier layer. The output layer of the neural network is a sigmoid function. Thus, y e,t is the probability that this most recent event history contains malicious activity.

Deep Reinforcement Learning. We next present a deep reinforcement learning model to control the antimalware engine's execution of an unknown file. The first task is to choose the type of reinforcement learning model for our problem. The main feature in our problem is a very large state space together with a small action space consisting of two actions A ∈ {continue, halt}. Considering the small action space, we prefer a value-based reinforcement learning technique which 

Design of States.:

The DRL model's state, which is illustrated in Figure 5 , contains three parts: the position (i.e., index) of the current event in the file ρ t , the current event ID, and the histogram of all the previous events. The current event position in the file will be used later to define the reward r t for the deep reinforcement learning model. We initially tried to use a one-hot encoding of the event ID, but found that using the event ID directly provides slightly better performance while reducing the size of the state. The event ID histogram captures the history of the events which have been observed so far. Design of Actions. The agent (i.e., antimalware engine) can perform two types of actions a t : continue, which is labeled as C, and halt, labeled as H. The action indicates whether the agent should continue or halt the execution of the file. In our deep reinforcement learning model, the selected action a t for state s t is inferred from the output of the neural network. As shown in the Figure 2 , the outputs of the deep neural network are the estimated action value function

Q C = Q * (s t , a t = C) = max π E[R t |a t = C, s t , π] for action C at state s t and Q H = Q * (s t , a t = H) = max π E[R t |a t = H, s t , π]

for action H at state s t . By comparing the two Q * (s t , a t ) values for actions H and C, the action with the larger Q value is selected and performed.

Design of Rewards. The reward r t at each state s t is designed based on two criteria: 1. We prefer for the DRL network to learn to halt emulation as quickly as possible. Therefore, shorter emulation sequence lengths are assigned a higher reward and longer sequence lengths are given a smaller reward. 2. The closer an event prediction is to the true label of the file, the larger the reward should be given at that state.

Based on the above two criteria, the reward is defined as

EQUATION

where y e,t is the event-based prediction generated by the most recent t f = 200 events, and L ∈ {0, 1} is defined as the true label of the training file. The decay factor β is chosen experimentally, and ρ t is the position of current event in the file.

DRL Training. where s t+1 are the neighbors of s t , and a t+1 are the corresponding actions generated by the neural network.

In Mnih et al. (2013) , experience replay is used to train the DRL model. Experience replay helps to alleviate the potential issues of non-stationary distributions and correlated data and is performed by randomly sampling the state pairs. Whenever one stochastic step is taken by the agent, the current state s t , obtained reward r t , action taken a t and next state s t+1 are combined as one agent experience set (s t ,r t ,a t ,s t+1 ), and pushed into the replay memory queue M . The reinforcement learning updates are performed in minibatches of size B RL , drawing from the replay memory randomly. The algorithm for training the DRL model using experience replay is provided in Appendix D.

We tested several different stochastic gradient descent optimization methods for training the DRL model and found that adadelta (Zeiler (2012)) performed best. Furthermore, the convergence of DRL is not always guaranteed. To help with the convergence, the sum of r t + γ should be within the range of [0,1]. It is important to note that we first train the event classifier, as well as the baseline file classifier in the improved inference model, in isolation prior to training the DRL model -the system is not trained in an end-to-end fashion. Thus, y e,t in the reward r t is generated by the pre-trained event classifier, and the reward function has the same value for the same event sequence. Otherwise, the DRL's reward function can become non-stationary.

Action State Model. The final block in Figure 2 is the action state model which generates the halting signal h t . The Q value signal is noisy and cannot be used directly to compute h t . The halting signal is a binary signal which is initialized to have a value of 0 and remains 0 for each e t until Q H > Q C for K consecutive events. At that point, the value of h t transitions to 1 and continues to maintain that value if any additional events are processed.

5. IMPROVED INFERENCE:

The purpose of the improved inference model in Figure 1 is used to generate a better DRL-based prediction, y RL,t , of whether or not an unknown file is malicious. The details of improved inference model are depicted in Figure 4 . This model has three inputs, namely the events generated by the file (e t ), the most recent event predictions (y e,t ), and the execution control signal (h t ). The most recent K values of y e,t are stored in the event prediction queue, which is another FIFO queue. After h t signals that the file execution has been halted, the improved file score model evaluates the event predictions in the queue to generate y RL,t .

The individual y e,t values, stored in the event prediction queue, are noisy and can be difficult to analyze. In some cases, only setting y RL,t to be the most recent y e,t value can lead to an incorrect prediction that the file is malicious. To overcome this issue, the improved inference model also employs an additional baseline file classifier to improve the accuracy of y RL,t . To accomplish this task, the initial t f (e.g., 200) events generated by the file are stored in the initial events buffer, and these are processed by the baseline file classifier to produce an initial prediction that the file is malicious, y f .

Baseline File Classifier. The baseline file classifier also utilizes the structure shown in Figure 3 and follows Athiwaratkun & Stokes (2017) . The input event subsequence in Figure 3 corresponds to the first t f events for each file stored in the initial events buffer (Figure 4 ). Here, t f is the same value which denotes the length of the event classifier's input event queue. An LSTM is used for the recurrent layer, and the classifier layer uses logistic regression for the file's prediction y f . Similar to the event classifier, y f is the initial estimate of the probability that the file is malicious based on the initial behavior of the file.

Improved File Score. We can now combine y f with the y e,t history stored in the event prediction queue to compute the final improved file classifier score, y RL,t . Since the initial estimates of the y e,t are noisy, we process the most recent K event predictions from the event prediction queue. Formally if h t is equal to 1,

EQUATION

where y e,i is the event classifier's prediction at step i, and y RL,t is the improved inference model's output, i.e., the prediction probability that the unknown file is malicious.

It is important to provide mathematical insight why (2) provides better performance compared to the baseline model which only uses a deep learning-based classifier to process the first t f events. The intuition is that our new model provides a closer prediction to the true label (e.g., malware or benign) compared to a single deep learning file classifier. Theorem 1. Let E st f [R t f ] be the expected reward obtained at the state s t f corresponding to event e t f , where t f is the fixed number of events used to train the baseline file classifier and is also the number of the most recent events used to train the event classifier. E s * [R * ] is the expected reward obtained at the halting state s * where s * is selected by the improved inference model when h t transitions from 0 to 1. If all models are fully trained and the decay factor β defined in reward is small enough,

E st f [R t f ] ≤ E s * [R * ] and |y * -L| ≤ 1 2

where y * ∈ [0, 1] is the predicted probability of a file being malicious using the improved inference model, and L is the true label of the given file. Then,

| y f -L |≥| y * -L | (3)

where y f is the predicted malware probability using a baseline file classifier.

This theorem is proved in Appendix E.

Evasion of adversarial learning-based attacks is another important aspect to consider. Evasion of the model discussed in Appendix F.

6. EXPERIMENTAL RESULTS:

We next present the results for the proposed neural malware control model. We first describe the datasets which were collected for our experiments. Next, we present the experimental setup. We then evaluate how quickly the DRL-based model halts the execution of a file. Finally, we compare the final prediction that the file is malicious or benign to the results from several baseline file classifiers.

Datasets. We collected a dataset of 75 thousand files which had been evaluated by a production antimalware engine. These files were equally split between the malware and benign classes. First, we discarded any files whose event sequences were shared between these two classes and which contained less than 50 events. Furthermore, we ensured that all the event sequences in the datasets were distinct. This requirement ensured that we did not overfit to one particular set of events. We then split the overall dataset into separate training, validation, and test sets with 50, 10, and 15 thousand files, respectively. Again, we maintained an equal split between the two classes for each of the individual datasets. The event and file classifiers were trained with the training and validation sets, while the DRL-model was trained with 2000 files from the training set. The results presented below are based on evaluating the model on the hold out test set. Setup. We implemented the proposed neural malware control model using Keras (Chollet et al. (2015) ) with Theano (Al-Rfou et al. ( 2016)) as the backend deep learning framework. Several hyperparameters were tuned on smaller datasets. The decay factor in the DRL model's reward function is γ = 0.01. The DRL model uses 3 hidden layers and is trained with a minibatch size B RL = 50. The replay memory is initialized with a size µ = 50000.

How does the DRL-based model's stopping performance compare to the antimalware engine's heuristics? The files in our dataset were collected by a production antimalware engine, and the number of events recorded for each file represents the performance of the heuristics employed by the engine to halt emulation. Thus, by measuring how often the DRL-based model halts execution prior to reaching the end of the file, we can compare the performance between our model and the engine's heuristics. In cases where the DRL model reaches the end of the file without halting execution, we can infer that the proposed model was not confident enough to make a decision, and the DRL based-model would have continued to execute the file.

The results of this evaluation are presented in Table 1 and depend on two values: the number of training files (i.e., epochs) N and the number of consecutive events where Q H > Q C denoted by K. The fraction of files where the DRL-model halts execution before the end of the file, α, is computed as: α = (Total number of early halted files)/(Total number of files).

We make two observations from the results presented in the table. First, the percentage of files whose execution is halted by the DRL model earlier than engine's heuristics continues to increase as the number of training file N increases. Better training allows the engine to halt execution earlier. Second, the percentage of files which are halted early decreases with K. The value of K is a proxy for the DRL model's confidence in the decision to halt the file's execution. It is not surprising that the execution of fewer files is halted early as we require more confidence (i.e., higher value of K) in the decision. Even so, the results show that over 91% of the files in the test set are halted early compared to the engine's heuristics after training with only 2000 files. This indicates that the engine's heuristics may be overly cautious when emulating a file. In addition, requiring less time for scanning an individual file leads to better performance when scanning all the files on the hard drive. A histogram which indicates the percentage of events which were executed before the DRL model halted execution is provided in Appendix G. Can DRL improve file classification? While the results in Table 1 indicate that emulation of the majority of the files can be stopped earlier than the heuristics employed in the engine, it is important to understand how early halting affects the detection performance. To measure this, we first compute the receiver operating characteristic (ROC) curves for a range of models in Figure 6 for the baseline file classifier depicted in Figure 3 . The models include LSTM Athiwaratkun & Stokes (2017) , RNN Pascanu et al. (2015) , gated recurrent unit (GRU) Athiwaratkun & Stokes (2017) , convolutional neural network (CNN) Kolosnjaji et al. (2016) and a simple, single hidden layer, feedforward neural network. We also include the echo state network (ESN) counterparts for the LSTM and GRU. All models use logistic regression (LR) for the classifier, because these slightly outperformed a shallow neural network for this dataset. All models use max-pooling as shown in Figure 3 . These results indicate that the performance of none of the models we investigated dominated all of the other models.

In particular, the ESN version of the GRU offers better performance and low false positive rates (FPRs) while the LSTM outperforms all other models above an FPR ≥ 1.2%.

We next compare the two best performing baseline file classifiers to the proposed DRL-based models for K ∈ {10, 15, 20} in Figure 7 . The figure clearly indicates that all the DRL-based models offer significantly better performance compared to the baseline file classifiers. In particular, the DRL-based model with K = 20 offers a relative improvement of 61.5% for the true positive rate (TPR) at an FPR of 1% compared to the GRU-ESN-based baseline file classifier. The relative improvement of the TPR is 65.7% at an FPR of 1% for the LSTM.

7. RELATED WORK:

Deep Reinforcement Learning. Conventional reinforcement learning has been widely studied in the fields of machine learning and system control for over three decades (Sutton (1984) ; Williams (1992); Littman (1994); Kaelbling et al. (1996) ; Sutton & Barto (1998)). Recently, Mnih et al. (2013; 2015) successfully applied deep neural network-based Q-learning (DQN) to playing a series of Atari games, by using a replay buffer to improve the system's convergence. Also, Silver et al. (2016; 2017) developed novel algorithms, by applying reinforcement learning to Monte Carlo tree search, to play the Go game and beat human Go masters. Progress has also been made in 2016)) deep reinforcement learning algorithms, in order to find better policies more efficiently and to deal with a continuous action space.

Despite the success of applying DRL to different types of games, no existing research has studied malware control and behavioral classification using reinforcement learning. The proposed neural malware control model is the first to apply a deep reinforcement learning-based algorithm for these tasks.

Deep Learning for Malware Classification. A number of authors have proposed using DNNs for malware classification tasks. Kephart (1994) employed a shallow neural network in the first paper on malware classification. Dahl et al. (2013) first investigated the use of a DNN for malware classification for dynamic analysis. Huang & Stokes (2016) proposed a DNN with multitask learning for dynamic analysis where the first task was binary (i.e., malware versus benign) and the second task was malware family classification. Saxe & Berlin (2015) proposed a DNN for the static analysis of malware. A separate line of research has investigated using recurrent models for malware classification. Pascanu et al. (2015) first proposed using recurrent neural networks and echo state networks for classifying malware sequences. Athiwaratkun & Stokes (2017) instead proposed an LSTM, a GRU, and a character-level CNN for the sequence classification. Kolosnjaji et al. (2016) used a CNN followed by an LSTM for this task.

8. CONCLUSION:

We present a novel, neural malware control model which learns when to halt the execution of an unknown file based on deep reinforcement learning. This model is the first to use deep reinforcement learning to protect customers from malware. Fast scanning is an important feature for users when installing software, reading emails with attachments, and searching a hard drive for files which were maliciously dropped during a drive-by download. Our results indicate that the proposed model halts execution earlier than a production antimalware engine for more than 91% of the files in the test set. More importantly, we show a relative improvement of over 61% in the true positive detection rate of malware at a false positive rate of 1% compared to a number of baseline malware classifiers reported in the literature. Thus, the proposed model offers significantly better protection with less delay.

malicious activity (i.e., cloaking). Based on our current datasets, our system may fail to detect any malware which successfully employs these cloaking mechanisms. end while 25: end for if the classifier is fully trained, i.e. y e,t f ≈ y e,t f +1 . Using these two approximations and substituting (7) into ( 6), ( 6) can be rewritten as

EQUATION

Since there are only two possible actions (halt, continue), the probability of remaining at a particular state p t at step t is either 0 or 1. If the system halts before t f events, p t f = p t f +1 = 1. Similarly, if the system continues until at least t f + 1 events, p t f = p t f +1 = 0. Assuming the system does not transition from continue to halt at event t

= t f , p t f + (1 -p t f )p t f +1 = 2p t f -p 2 t f = p t f

. Under this weak assumption, (8) can be rewritten as

EQUATION

Similarly, E s * [R * ], the expected rewards obtained at the halting state s * chosen by the DRL-based model, can be calculated as

EQUATION

where p * is the probability of remaining at state s * , y * is the predicted probability generated by the event classifier using the most recent t f events before the halting state s * , ρ * is the position of the event contained in state s * , and E s * [R * ] is the expected reward at state s * which is the next state after s * . Since s * is an absorbing (i.e., terminal) state, p * = 1. Thus,

EQUATION

and the expected reward at s * is larger or equal than those at any other states

EQUATION

Substituting ( 9) and ( 11) into (12), p t f ((0.5 -|y e,t f -L|)e -βρt f ) + (1 -p t f )C t f +1 ≤ (0.5 -|y * -L|)e -βρ * .

(13)

Recalling p t f is equal to either 0 or 1 and first considering the case when p t f is equal to 1, (13) can be further rewritten as (0.5 -|y e,t f -L|)e -βρt f ≤ (0.5 -|y * -L|)e -βρ * 0.5 -|y e,t f -L| ≤ (0.5 -|y * -L|)e β(ρt f -ρ * )

(14)

Recall that the decay factor β is defined to be positive. If ρ t f < ρ * , then 0 < e β(ρt f -ρ * ) < 1, and ( 14) can be rewritten as (16)

By choosing small enough β such that e β(ρt f -ρ * ) → 1, the second term in ( 16) is approaching zero, then |y e,t f -L| ≥ |y * -L|. To prove it strictly, choose a β satisfying (18)

The proceeding derivation was for the case when p t f = 1. Following (5) for the case when p t f = 0, we have

E st f [R t f ] = E st f +1 [R t f +1 ]. Similarly we have E st f +1 [R t f +1 ] = E st f +2 [R t f +2

] since p t f +1 = 0, and so on. Thus, E st f [R t f ] = E st f +n [R t f +n ] where s t f +n is the first halting state after s t f , such that p t f +n is equal to 1. Then we can substitute t f by t f + n in our proof in order to show the same result that |y e,t f -L| ≥ |y * -L| for the case when p t f = 0.

While we have now shown that |y e,t f -L| ≥ |y * -L| for the event classifier, we need to consider the prediction probability of the baseline file classifier. The predicted probability y e,t f generated by the event classifier using the first t f events is an approximation of y f , which is the predicted probability generated by the baseline file classifier also using the first t f events, i.e. y e,t f ≈ y f , hence we have

EQUATION

)

It can be concluded that the DRL predicted probability y * is closer to the true label than the baseline file classifier's predicted probability y f .

APPENDIX F: EVASION:

Recently, researchers have begun to investigate adversarial attacks on deep reinforcement learning models. Since the proposed DRL-based system would be run on a user's computer or cell phone, it is particularly vulnerable to adversarial attacks. The basic strategy is to destabilize the DRL network by

:



Figures and Tables:

Figure2: Details of the execution control model in Figure1. In both figures, the input is e t , and the outputs are h t and y e,t .

Figure 5: State representation, s t , for the DRL model.

Figure 6: Performance of several recently proposed recurrent models for the baseline file classifier.

improving value-based (Van Hasselt et al. (2016); Wang et al. (2016)), policy gradient (Schulman et al. (2015); Duan et al. (2016); Gu et al. (2017); O'Donoghue et al. (2017)), and actor-critic (Lillicrap et al. (2015); Mnih et al. (

-|y e,t f -L| ≤ 0.5 -|y * -L| |y e,t f -L| ≥ |y * -L|. (15) If ρ t f > ρ * , then e β(ρt f -ρ * ) > 1, and (14) can be rewritten as 0.5 -|y e,t f -L| ≤ (0.5 -|y * -L|)e β(ρt f -ρ * ) ≤ 0.5 -|y * -L| + (e β(ρt f -ρ * ) -1)(0.5 -|y * -L|).

(e β(ρt f -ρ * ) -1)(0.5 -|y * -L|) ≤ 1 2 ((0.5 -|y e,t f -L|) -(0.5 -|y * -L|)). (17)By substituting (17) into (16), we can derive that |y e,t f -L| ≥ |y * -L|.Given that |y * -L| ≤ 1 2 , (17) can provide an upper bound of β as(e β(ρt f -ρ * ) -1)(0.5 -|y * -L|) ≤ 1 2 ((0.5 -|y e,t f -L|) -(0.5 -|y * -L|)) (e β(ρt f -ρ * ) -1) ≤ (|y * -L| -|y e,t f -L|) 2(0.5 -|y * -L|) e β(ρt f -ρ * ) ≤ 1 + (|y * -L| -|y e,t f -L|) 1 -2|y * -L| β ≤ 1 ρ t f -ρ * ln(1 + (|y * -L| -|y e,t f -L|)1 -2|y * -L| ).



Table content: <table><tr><td/><td>Execution Control</td></tr><tr><td>DRL</td><td>Signal 𝒉𝒕</td></tr><tr><td/><td>𝒚𝒆,𝒕</td></tr><tr><td>𝒚𝒆,𝒕</td><td/></tr><tr><td/><td>Improved File</td></tr><tr><td/><td>Classifier Score</td></tr><tr><td/><td>𝒚𝑹𝑳,𝒕</td></tr><tr><td colspan="2">Figure 1: Deep reinforcement learning system</td></tr><tr><td colspan="2">for halting the execution of an unknown file and</td></tr><tr><td>improved malware classification.</td><td/></tr></table>



Table content: <table/>

3: Structure of the event classifier and the baseline file classifier. Details of the improved file inference model in Figure 1. The inputs are e t , y e,t , and h t , and the output is y RL,t in both figures.

Table content: <table><tr><td/><td colspan="2">Execution Control Model</td><td/><td/><td/></tr><tr><td/><td/><td/><td/><td>𝒚𝒆,𝒕</td><td>𝒉𝒕</td></tr><tr><td/><td/><td/><td colspan="2">Event Prediction</td><td/></tr><tr><td/><td/><td/><td/><td>Queue</td><td/></tr><tr><td>File Events, e t</td><td>Initial Buffer Events</td><td>Baseline File Classifier</td><td>𝒚𝒇</td><td>Improved File Score Model</td><td>𝒚𝑹𝑳,𝒕</td></tr><tr><td/><td/><td colspan="3">Improved Inference Model</td><td/></tr><tr><td>Figure 4:</td><td/><td/><td/><td/><td/></tr></table>

To train this neural network-based estimator, we use an l 2 loss function defined asL(θ t ) = E st [( Q(s t , a t |θ t ) -Q(s t , a t |θ t )) 2 ] where Q(s t , a t |θ t ) is an estimate of Q(s t , a t |θ t ).Q(s t , a t |θ t ) is computed using the current state reward r t together with its neighbors' estimations from the neural network in an iterative manner, i.e., Q(s t , a t |θ t ) = r t + γ max at+1 Q(s t+1 , a t+1 |θ t )

Table content: <table/>

The fraction of files, α, in % where emulation is halted earlier by the proposed deep reinforcement learning model compared to heuristics used by the antimalware engine.

Table content: <table><tr><td/><td>K=10</td><td>K=15</td><td>K=20</td></tr><tr><td>N =30</td><td>71.5%</td><td>64.1%</td><td>58.3%</td></tr><tr><td>N =200</td><td>82.9%</td><td>75.2%</td><td>69.2%</td></tr><tr><td>N =2000</td><td>98.2%</td><td>95.1%</td><td>91.3%</td></tr></table>

Peer review (iclr2019):
 This paper attempts to train a predictor of whether software is malware. Previous studies have emulated potential malware for a fixed number of executed instructions, which risks both false negatives (haven’t yet reached the dangerous payload) and false positives (malware signal may be lost amidst too many other operations). This paper proposes using deep reinforcement learning over a limited action space: continue executing a program or halt, combined with an “event classifier” which predicts whether individual parts of the program consist of malware. The inputs at each time step are one of 114 high level “events” which correspond to related API invocations (e.g. multiple functions for creating a file). One limitation seems to be that their dataset is limited only to events considered by a "production malware engine", so their evaluation is limited only to the benefit of early stopping (rather than continuing longer than the baseline malware engine). They evaluate a variety of recurrent neural networks for classifying malware and show that all significantly underperform the “production antimalware engine”. Integrating the event classifier within an adaptive execution control, trained by DQN, improves significantly over the RNN methods. It might be my lack of familiarity with the domain but I found this paper very confusing. The labeling procedure (the "production malware engine”) was left entirely unspecified, making it hard to understand whether it’s an appropriate ground-truth and also whether the DRL model’s performance is usable for real-world malware detection. Also, the baseline models used an already fairly complicated architecture (Figure 3) and it would have been useful to see the performance of simple heuristics and simpler models.