# EvadeRL: Evading PDF Malware Classifiers with Deep Reinforcement Learning

Authors: Zhengyang Mao; Zhiyang Fang; Meijin Li; Yang Fan
Journal: Security and Communication Networks
 (2022)

---

## Abstract

With the growing popularity of information digitization and the advancement of executable file detection technology, PDF has emerged as an important carrier of malicious documents. Despite the improved efficacy of machine learning-based classifiers in detecting PDF malware, adversaries have proposed a variety of countermeasures to evade detection, such as generating adversarial examples. In contrast to other peer works attempting to expose the vulnerability of learning-based detection models, this work addresses the deficiencies of existing research by pointing out that the stochastic manipulations they applied may be highly computationally demanding. This work proposed EvadeRL, a general framework for automatically generating adversarial examples based on double deep Q-Network. The details of EvadeRL are briefly described as follows. First, the EvadeRL agent chooses a series of actions to modify the given PDF files and uses the classification results, as well as observations returning from the environment, to calculate the approximate value of each action. Second, through the interaction of the agent and the environment, the experiences gained are stored to train the decision network. Finally, the agent can generate adversarial examples against the target detector by taking the optimal behaviors after training. This study also contributes to the sustainability of evasion attacks by online fine-tuning; to the best of our current knowledge, this is the first study in the field that focuses on evolving malware. The experiments reveal that EvadeRL obtains a high evasion rate against PDF malware detectors and outperforms other approaches in terms of execution cost, robustness against hardened detectors, and sustainability against evolving malware and detectors.


---

## Body

## 1. Introduction

Machine learning (ML) has been adopted in an extensively broad variety of application domains, including natural language processing, computer vision, and security tasks. Considerable research efforts have been devoted to applying ML in security contexts, especially for the detection of network intrusion [1], classification or clustering of malware families [2, 3], and detection of nonexecutable documents such as portable document format (PDF) malware. Due to the steady improvement of robustness in the detection of executable malware, nonexecutable files, such as PDF, have increasingly been utilized as infection vectors by adversaries [4]. According to Internet Security Threat Report published by Symantec in 2019 [5], nonexecutable documents like PDF and Microsoft Office files accounted for almost half of all malicious e-mail attachments. In recent years, there has been a surge in the study on malware detection for PDF files using ML [6–9] to lower the risk of threat.While ML has contributed a lot and becomes the dominating force in malware detection, it is noteworthy that learning-based systems are vulnerable to potential attacks [10]. In particular, attackers can generate intentionally manipulated adversarial examples to deceive learning-based classifiers. Recent research has demonstrated that a minor perturbation to the input data can mislead the ML model to produce incorrect classification results at test time [11–13]. Evasion attacks have the potential to diminish the effectiveness of ML models and have a wider impact across all application domains that use ML [14]. Thus, the robustness of these methods is urged to be improved. One feasible way to achieve this goal is to harden the model by incorporating adversarial examples in the model training process [15]. For this reason, giant efforts have been made to generate adversarial examples against ML models to expose their vulnerabilities for further optimization. To evaluate the robustness of the ML-based PDF malware classifier, this study adopts deep reinforcement learning (DRL) to generate adversarial examples and evaluate it against black-box PDF malware classifiers.Adversarial example generation techniques have escalated as ML techniques for malicious PDF detection become increasingly sophisticated. Šrndić and Laskov [16] studied the preliminary form of evasion techniques, known as Mimicry, in which the adversaries have a relatively high level of knowledge about the target systems. Biggio et al. [17] proposed a method using gradient descent attack and evaluated its efficacy in different assault scenarios. Xu et al. [18] proposed an approach called EvadeML, which is based on genetic programming and operates in a more realistic evasion scenario in which adversaries only have limited knowledge of the target classifiers. Dang et al. [19] presented a method named EvadeHC using the mountain climbing algorithm, which was claimed to be more time-efficient while preserving a high evasion rate. Recent studies done by Bae et al. [20] and Li et al. [21] applied generative adversarial network (GAN) to bypass detectors and achieved better performance than previous techniques.Despite the remarkable success of the aforementioned researches, it has been noted that both EvadeML and EvadeHC have deficiencies that the stochastic manipulations they applied may be highly computationally demanding [20]. And conceivably, random mutations (i.e., inserting, deleting, and replacing) [18] to the structure make it challenging to maintain the maliciousness of the original files. Hence, a general framework named EvadeRL using double deep Q-Network (DDQN) [22] to evade the black-box PDF detector is proposed in this work. EvadeRL framework comprises the target classifier and an agent set up to confront the PDF detector, capable of generating adversarial examples automatically through the interaction with the environment. In EvadeRL, the agent continuously inspects the specific observation space for any given PDF sample and determines the optimal choice of the maliciousness-preserving action until the modified malicious PDF sample is misclassified by the malware classifier. Through the interactions between the agent and the environment, experiences composed of states, actions, and rewards are stored in the replay buffer and utilized to train the agent’s decision network. After training, the agent manages to select the appropriate sequence of modifications to given PDF files, and the results show that EvadeRL can achieve a high evasion rate against state-of-the-art detectors. Given that the target detectors and the features of original files are irrelevant to the proposed framework, it can be extended by reworking mutation actions to evade other malware detectors such as PE files and Android APK files. Moreover, this study focuses on the continuously evolving malware samples and malware detectors [39, 40] that may lead to the deterioration of evasion techniques. EvadeRL leverages a two-stage training workflow, that is, offline pretraining and online fine-tuning, to gain sustainability against evolving PDF malware.The contributions of this paper can be summarized as follows:(i)
EvadeRL is proposed for automatically generating adversarial examples against target classifiers, which can find an optimal order of mutations to the PDF files without crashing the file’s initial malicious functionality and accomplish the black-box attack(ii)
EvadeRL leverages online fine-tuning to obtain sustainability, allowing for rapid policy improvement against evolving malware samples and detectors by retraining with a combination of offline data and online experience(iii)
Thorough experiments were carried out to evaluate the effectiveness of EvadeRL against PDF malware detectors, which revealed that it could achieve approximately the same evasion rate as the existing evasion approaches while outperforming them in terms of execution cost, robustness, and sustainabilityThe rest of this paper is organized as follows: Section2 introduces an overview of the related work. Section 3 describes the proposed EvadeRL model in detail. Section 4 presents the explicit experimental setup and numerical simulation results. Section 5 concludes the paper.


## 2. Related Work

### 2.1. PDF Structure

A typical structure of a PDF file contains four parts: header, body, cross-reference table, and trailer, and it is shown in Figure1. The header is used to identify the version of a PDF file. The body consists of a sequence of objects, including embedded data like images, executable codes, and stream objects which are unlimited in length. The cross-reference table is a list of offsets that can be used to index all the objects in the file. It is responsible for informing the PDF parsers from where to begin the parsing procedure [23]. The trailer contains descriptions of some essential elements, such as where to find the cross-reference table and other basic objects.Figure 1
An example of the PDF file structure.Due to the complex yet explicit structure of PDF files [24], adversaries have various ways to inject malicious content into the PDF files, such as JavaScript-based, ActionScript-based, and file-embedding attacks [25]. Among them, injecting JavaScript codes is the most common way to perform attacks [26], with the primary purpose of redirecting the execution flow to the embedded malicious JavaScript code.


### 2.2. Target PDF Classifiers

Smutz and Stavrou [27] presented PDFRate, a system for detecting malicious PDF files. PDFRate implemented a parser that extracts metafeatures from PDF documents, including metadata such as title and creation data, as well as structural information like the number of objects, the average length of stream objects, and so forth. With a total of 202 features chosen, PDFRate utilizes the Random Forest algorithm for classification. PDFRate is trained on randomly selected samples from the Contagio dataset [28], and the results demonstrate that it can achieve over 99% TPR with FPR lower than 0.2%.Corum et al. [29] proposed a novel approach to detect malicious PDF. The PDF files are first converted to grayscale images using image visualization techniques, and various image features are then extracted to be classified by ML models like Random Forest, Decision Tree, and KNN. The proposed method was used to stage against reverse Mimicry attack, and the results revealed that it has better robustness. Li et al. [30] improved the previous detector Hidost [7] by using active learning to boost training. The newly proposed method focused on training with a small size of training set while still maintaining the performance of detection.


### 2.3. Evasion Techniques of Generating Adversarial Examples

The previous evasion approaches can be categorized using a taxonomy proposed by Maiorca et al. [31], which is shown in Table 1. Following the proposed categorization depicted in Table 1, the related work can be summarized as follows.Table 1
The categories of the evasion approaches.
Classification basisTypeDescriptionThe family of evasion techHeuristic-based attackIt performs reasonable mutations which are believed to cause misclassification of targeted classifiersOptimization-based attackAn optimization problem is often formulated to find the solution to generate evasive examples by fine-grained modificationsThe knowledge of the learning modelWhite-box attackAttack with a complete understanding of the detection modelGray-box attackAttack with the knowledge of feature representationBlack-box attackAttack with no information about the detection model


#### 2.3.1. White-Box Heuristic-Based Attacks

Heuristic-based strategies are generally less effective than optimization-based ones, and there are two main strategies: the Mimicry and reverse Mimicry attack [32] in this category. Šrndić and Laskov [16] deceived PDFRate by Mimicry attack, which attempted to modify the feature vector of a malicious sample to make it as close as possible to a benign one. Moreover, the target feature vector was further transformed into an evasive PDF file by content injection. The authors performed the Mimicry in different scenarios where the adversary had varying levels of knowledge. The results showed that the evasion rates were over 75% in all three scenarios, and they also revealed that it was possible to reach a considerable evasion rate by merely knowing the feature sets used by the classifiers. In addition, with perfect knowledge of the target system, a similar approach [27] managed to decrease the accuracy of the classifier by over 20% by mutating only six key features [33–35].


#### 2.3.2. Black-Box Optimization-Based Attacks

Optimization-based attacks are based on an idea to minimize the formula presented by Biggio et al. [17] as(1)argminx′fx′−λgx′,(2)s.t.x≤x′,andx−x′1≤ε,where the malicious vector and adversarial vector are denoted by x and x′, respectively. The main idea of (1) is to minimize the classification score fx′ and maximize gx′, which is a measure of how similar the altered sample is to the benign one.Xu et al. [18] presented EvadeML, a black-box attack model based on genetic programming. It aimed to create evasive variants by performing stochastic mutations including insertion, deletion, and replacement of the content. To begin with, a population of variants is generated through random operations and then checked by an oracle to ensure that the original maliciousness remains. The successful variants are then employed to create the next generation of the population. EvadeML has been staged against PDFRate and Hidost, and the results indicated that the method successfully reached a 100% evasion rate. However, the proposed method was extensively computationally demanding and it took approximately six days to generate all the evasion variants.Dang et al. [19] proposed EvadeHC to speed up the evasion procedure. The authors first introduced the notions of malice-flipping distance and reject-flipping distance, which represent the morphing steps required for a malicious PDF to lose maliciousness or to be recognized as a benign one, respectively. Thus, the authors adopted the Hill-Climbing algorithm to search for reject-flipping samples that have not reached their malice-flipping distance. The proposed method was evaluated on Hidost and PDFRate and proven to be very effective and more time-efficient than EvadeML.Li et al. [21] presented fvGAN to generate adversarial feature vectors in the feature space. The process begins with extracting the characteristics of the original malicious PDF files; each PDF file is represented as a 135-dimensional vector based on the retrieved features. These vectors are then fed into the fvGAN module, which generates adversarial PDF malware in the feature space. Finally, the created adversarial vectors are transformed into actual malicious PDF files. The experimental results showed that fvGAN can marginally outperform the Mimicry attack in terms of evasion rates.The recent development of evasion techniques using RL has attracted much attention. A pioneering study proposed by Anderson et al. [36] presented a generic black-box attack based on RL to evade Windows PE machine learning models. Based on previous work, Fang et al. [37] constructed a DRL adversarial model which could mislead the targeted ML model to misclassify one malware into a specific category.Based on the previous work, this paper focuses on improving the following aspects: (1) By utilizing DRL to build the EvadeRL model, the agent learns to automatically generate adversarial examples with optimal actions but not stochastic modification. (2) Each mutation action is guaranteed not to corrupt the original file and cause loss of maliciousness. Thus, the training procedure is more time-efficient without constantly interacting with an oracle to test the maliciousness of mutated files. (3) The previous work ignored the problem of drifting malware and evolving detectors. Therefore, EvadeRL addresses the problem of sustainability by retraining with the combination of offline data and online experience to fine-tune the network.


## 2.1. PDF Structure

A typical structure of a PDF file contains four parts: header, body, cross-reference table, and trailer, and it is shown in Figure1. The header is used to identify the version of a PDF file. The body consists of a sequence of objects, including embedded data like images, executable codes, and stream objects which are unlimited in length. The cross-reference table is a list of offsets that can be used to index all the objects in the file. It is responsible for informing the PDF parsers from where to begin the parsing procedure [23]. The trailer contains descriptions of some essential elements, such as where to find the cross-reference table and other basic objects.Figure 1
An example of the PDF file structure.Due to the complex yet explicit structure of PDF files [24], adversaries have various ways to inject malicious content into the PDF files, such as JavaScript-based, ActionScript-based, and file-embedding attacks [25]. Among them, injecting JavaScript codes is the most common way to perform attacks [26], with the primary purpose of redirecting the execution flow to the embedded malicious JavaScript code.


## 2.2. Target PDF Classifiers

Smutz and Stavrou [27] presented PDFRate, a system for detecting malicious PDF files. PDFRate implemented a parser that extracts metafeatures from PDF documents, including metadata such as title and creation data, as well as structural information like the number of objects, the average length of stream objects, and so forth. With a total of 202 features chosen, PDFRate utilizes the Random Forest algorithm for classification. PDFRate is trained on randomly selected samples from the Contagio dataset [28], and the results demonstrate that it can achieve over 99% TPR with FPR lower than 0.2%.Corum et al. [29] proposed a novel approach to detect malicious PDF. The PDF files are first converted to grayscale images using image visualization techniques, and various image features are then extracted to be classified by ML models like Random Forest, Decision Tree, and KNN. The proposed method was used to stage against reverse Mimicry attack, and the results revealed that it has better robustness. Li et al. [30] improved the previous detector Hidost [7] by using active learning to boost training. The newly proposed method focused on training with a small size of training set while still maintaining the performance of detection.


## 2.3. Evasion Techniques of Generating Adversarial Examples

The previous evasion approaches can be categorized using a taxonomy proposed by Maiorca et al. [31], which is shown in Table 1. Following the proposed categorization depicted in Table 1, the related work can be summarized as follows.Table 1
The categories of the evasion approaches.
Classification basisTypeDescriptionThe family of evasion techHeuristic-based attackIt performs reasonable mutations which are believed to cause misclassification of targeted classifiersOptimization-based attackAn optimization problem is often formulated to find the solution to generate evasive examples by fine-grained modificationsThe knowledge of the learning modelWhite-box attackAttack with a complete understanding of the detection modelGray-box attackAttack with the knowledge of feature representationBlack-box attackAttack with no information about the detection model


### 2.3.1. White-Box Heuristic-Based Attacks

Heuristic-based strategies are generally less effective than optimization-based ones, and there are two main strategies: the Mimicry and reverse Mimicry attack [32] in this category. Šrndić and Laskov [16] deceived PDFRate by Mimicry attack, which attempted to modify the feature vector of a malicious sample to make it as close as possible to a benign one. Moreover, the target feature vector was further transformed into an evasive PDF file by content injection. The authors performed the Mimicry in different scenarios where the adversary had varying levels of knowledge. The results showed that the evasion rates were over 75% in all three scenarios, and they also revealed that it was possible to reach a considerable evasion rate by merely knowing the feature sets used by the classifiers. In addition, with perfect knowledge of the target system, a similar approach [27] managed to decrease the accuracy of the classifier by over 20% by mutating only six key features [33–35].


### 2.3.2. Black-Box Optimization-Based Attacks

Optimization-based attacks are based on an idea to minimize the formula presented by Biggio et al. [17] as(1)argminx′fx′−λgx′,(2)s.t.x≤x′,andx−x′1≤ε,where the malicious vector and adversarial vector are denoted by x and x′, respectively. The main idea of (1) is to minimize the classification score fx′ and maximize gx′, which is a measure of how similar the altered sample is to the benign one.Xu et al. [18] presented EvadeML, a black-box attack model based on genetic programming. It aimed to create evasive variants by performing stochastic mutations including insertion, deletion, and replacement of the content. To begin with, a population of variants is generated through random operations and then checked by an oracle to ensure that the original maliciousness remains. The successful variants are then employed to create the next generation of the population. EvadeML has been staged against PDFRate and Hidost, and the results indicated that the method successfully reached a 100% evasion rate. However, the proposed method was extensively computationally demanding and it took approximately six days to generate all the evasion variants.Dang et al. [19] proposed EvadeHC to speed up the evasion procedure. The authors first introduced the notions of malice-flipping distance and reject-flipping distance, which represent the morphing steps required for a malicious PDF to lose maliciousness or to be recognized as a benign one, respectively. Thus, the authors adopted the Hill-Climbing algorithm to search for reject-flipping samples that have not reached their malice-flipping distance. The proposed method was evaluated on Hidost and PDFRate and proven to be very effective and more time-efficient than EvadeML.Li et al. [21] presented fvGAN to generate adversarial feature vectors in the feature space. The process begins with extracting the characteristics of the original malicious PDF files; each PDF file is represented as a 135-dimensional vector based on the retrieved features. These vectors are then fed into the fvGAN module, which generates adversarial PDF malware in the feature space. Finally, the created adversarial vectors are transformed into actual malicious PDF files. The experimental results showed that fvGAN can marginally outperform the Mimicry attack in terms of evasion rates.The recent development of evasion techniques using RL has attracted much attention. A pioneering study proposed by Anderson et al. [36] presented a generic black-box attack based on RL to evade Windows PE machine learning models. Based on previous work, Fang et al. [37] constructed a DRL adversarial model which could mislead the targeted ML model to misclassify one malware into a specific category.Based on the previous work, this paper focuses on improving the following aspects: (1) By utilizing DRL to build the EvadeRL model, the agent learns to automatically generate adversarial examples with optimal actions but not stochastic modification. (2) Each mutation action is guaranteed not to corrupt the original file and cause loss of maliciousness. Thus, the training procedure is more time-efficient without constantly interacting with an oracle to test the maliciousness of mutated files. (3) The previous work ignored the problem of drifting malware and evolving detectors. Therefore, EvadeRL addresses the problem of sustainability by retraining with the combination of offline data and online experience to fine-tune the network.


## 2.3.1. White-Box Heuristic-Based Attacks

Heuristic-based strategies are generally less effective than optimization-based ones, and there are two main strategies: the Mimicry and reverse Mimicry attack [32] in this category. Šrndić and Laskov [16] deceived PDFRate by Mimicry attack, which attempted to modify the feature vector of a malicious sample to make it as close as possible to a benign one. Moreover, the target feature vector was further transformed into an evasive PDF file by content injection. The authors performed the Mimicry in different scenarios where the adversary had varying levels of knowledge. The results showed that the evasion rates were over 75% in all three scenarios, and they also revealed that it was possible to reach a considerable evasion rate by merely knowing the feature sets used by the classifiers. In addition, with perfect knowledge of the target system, a similar approach [27] managed to decrease the accuracy of the classifier by over 20% by mutating only six key features [33–35].


## 2.3.2. Black-Box Optimization-Based Attacks

Optimization-based attacks are based on an idea to minimize the formula presented by Biggio et al. [17] as(1)argminx′fx′−λgx′,(2)s.t.x≤x′,andx−x′1≤ε,where the malicious vector and adversarial vector are denoted by x and x′, respectively. The main idea of (1) is to minimize the classification score fx′ and maximize gx′, which is a measure of how similar the altered sample is to the benign one.Xu et al. [18] presented EvadeML, a black-box attack model based on genetic programming. It aimed to create evasive variants by performing stochastic mutations including insertion, deletion, and replacement of the content. To begin with, a population of variants is generated through random operations and then checked by an oracle to ensure that the original maliciousness remains. The successful variants are then employed to create the next generation of the population. EvadeML has been staged against PDFRate and Hidost, and the results indicated that the method successfully reached a 100% evasion rate. However, the proposed method was extensively computationally demanding and it took approximately six days to generate all the evasion variants.Dang et al. [19] proposed EvadeHC to speed up the evasion procedure. The authors first introduced the notions of malice-flipping distance and reject-flipping distance, which represent the morphing steps required for a malicious PDF to lose maliciousness or to be recognized as a benign one, respectively. Thus, the authors adopted the Hill-Climbing algorithm to search for reject-flipping samples that have not reached their malice-flipping distance. The proposed method was evaluated on Hidost and PDFRate and proven to be very effective and more time-efficient than EvadeML.Li et al. [21] presented fvGAN to generate adversarial feature vectors in the feature space. The process begins with extracting the characteristics of the original malicious PDF files; each PDF file is represented as a 135-dimensional vector based on the retrieved features. These vectors are then fed into the fvGAN module, which generates adversarial PDF malware in the feature space. Finally, the created adversarial vectors are transformed into actual malicious PDF files. The experimental results showed that fvGAN can marginally outperform the Mimicry attack in terms of evasion rates.The recent development of evasion techniques using RL has attracted much attention. A pioneering study proposed by Anderson et al. [36] presented a generic black-box attack based on RL to evade Windows PE machine learning models. Based on previous work, Fang et al. [37] constructed a DRL adversarial model which could mislead the targeted ML model to misclassify one malware into a specific category.Based on the previous work, this paper focuses on improving the following aspects: (1) By utilizing DRL to build the EvadeRL model, the agent learns to automatically generate adversarial examples with optimal actions but not stochastic modification. (2) Each mutation action is guaranteed not to corrupt the original file and cause loss of maliciousness. Thus, the training procedure is more time-efficient without constantly interacting with an oracle to test the maliciousness of mutated files. (3) The previous work ignored the problem of drifting malware and evolving detectors. Therefore, EvadeRL addresses the problem of sustainability by retraining with the combination of offline data and online experience to fine-tune the network.


## 3. EvadeRL Framework

### 3.1. Model Structure

The general structure of RL consists of an agent, an environment, a state space, an action space, and the reward. An appropriate definition of these elements is a necessity when using RL to construct a model to generate adversarial examples. Figure2 shows the model architecture of EvadeRL. EvadeRL uses DDQN [22] as its learning algorithm to reduce overestimation by decomposing the max operation in the target into action selection and action evaluation. DDQN evaluates the greedy policy according to the decision network Q but uses the target network Q∗ to estimate its value. The agent chooses an action to modify the given PDF file based on the current knowledge, the feature extractor and detector in the environment determine the state and classification result of the modified file, and both the state and reward are then returned to the agent afterward. Through the continuous interaction of the agent and the environment, the experience gained each time-step is stored in the replay buffer for the update of Q. The agent’s ultimate purpose is to choose actions in a fashion that succeeds in generating adversarial examples and maximizes the reward.Figure 2
The architecture of EvadeRL.The environment is comprised of a feature extractor and an integrated PDF detector; the former is responsible for observing the sample state, and the latter is bound to determine the reward to any given PDF example by the classification result. The environment uses the reward and the next state as feedback to the agent. The agent chooses mostly random actions to mutate PDF files according to theε-greedy policy at the early episodes. As the training process progresses, the agent uses the observation state and the reward returned by the environment to compute the state-action value and update the parameters of Q. Thus, the action-choosing policy of the agent can be more targeted by always choosing the optimal action according to (3), where st is the current state, θ represents the parameters of the decision network, a is a set of actions, and at denotes the best action the agent should choose. The action space only comprises different kinds of content insertions into the PDF files which are guaranteed not to crash the malicious functions and the original formats of them. The details of the EvadeRL model are elaborated as follows:(3)at=argmaxQst,a;θ.


### 3.2. Environment

The PDF files are mapped to the memory and exist in the environment as byte sequences. The raw bytes, however, are not appropriate for the agent to observe. In order to represent the current state of the PDF sample more accurately, the byte sequence is transformed into the form of a feature vector. Any irrelevant features should be removed from the feature set for the agent to better observe the environment. To this end, the initial observation state is defined as the set of features that will be influenced by the actions. After each action is taken, the observation state will change according to (4), where cntai represents how many times action ai has been taken and λ is a constant, while stfi denotes the observation space of feature fi at time-step t. Following the equation above, the agent is sensitive to the changing states of files and the situation where a single action is overfrequently taken can be avoided.(4)stfi=st−1fi−λcntai.


### 3.3. Action Space

The action space is specified by two basic rules: every action of mutation is effective in deceiving the target classifier, while the modification must not damage the original function of files. In addition, designing an action space containing rich actions in the problem scope of RL may lead to a difficult training process. Thus, a relatively lean action space is defined asA = {META, IF, IB, IOAS, MID}. As is shown in Figure 3, the modification methods inject contents only between the cross-reference table and the trailer without removing or modifying the existing elements. The injected content is ignored by PDF readers but can be discovered by the PDFRate detector.Figure 3
Modify by injecting contents between the cross-reference table and the trailer.The following are detailed descriptions of the actions: (1) META: Modify the metadata in the PDF files, which is general information such as the document’s title, author, and creation dates. In particular, both uppercase and lowercase letters along with digits and separators are appended to the specific metadata, including author, creator, producer, subject, and title. (2) IF: Insert a new type of font to the PDF file, which defines glyphs for a particular character set. (3) IR: Insert rectangles that are used to describe locations on a page and bounding boxes for a variety of objects. A rectangle is written as an array of four numbers giving the coordinates of a pair of diagonally opposite corners. Thus, an array shaped like [llx,lly,urx,ury] is inserted to represent a rectangle. (4) IOAS: Insert objects and stream objects. An object starts with the keyword “obj” and ends with “endobj,” while the keywords for stream objects are “stream” and “endstream.” This action is performed by inserting keywords and random content. (5) MID: Modify the identifier of the PDF file. The value of the ID field is an array of two strings, each of which has 32 letters and digits. If the provided PDF file does not contain any IDs, two identifiers are inserted into the trailer dictionary. In other circumstances, IDs are changed at random.


### 3.4. Reward

The reward is a scalar feedback signal from the environment that indicates how well the agent performs on a specific step. Since our goal is to force misclassification of the target detector, the intuition is to use its classification outcomes as the reward. The target detector is integrated into EvadeRL as a black-box detector, which only returns true or false for the input PDF files. The reward is defined as either a positive number to indicate the agent has succeeded in forcing misclassification after several actions or zero if this goal is not achieved and a maximum of steps have been taken.


### 3.5. Agent

The agent uses the neural network to estimate the value of different actions in a certain state and follow the policy of always taking the most valuable action at each state. There are two neural networks, that is, the decision networkQ and target network Q∗, which share the same network structure but serve different purposes. To make the training procedure stable, the parameter θ of Q is updated every step to fit Q∗, while θ− of Q∗ is only updated every C step and is held fixed between individual updates [38].In the context of evading the PDF detector, the decision network takes the observation state as input and then produces the next mutation action the agent will take. As for the output layer of the network, it is set to action-index in the action setA. In EvadeRL, the output dimension equals 5, which refers to the total dimension of the action space. The network uses both Q and Q∗ to compute the loss function, which is defined in (5), in which θt and θt− are the parameters of Q and Q∗. The parameters of the Q network are then updated by backpropagation using gradient descent, which is shown in (6).(5)Lw=ΕRt+1+γQSt+1,argmaxaSt+1,a;θt,θt−︸Target−QSt,at;θt2,(6)θt+1=θt+αRt+1+γQSt+1,argmaxaSt+1,a;θt,θt−−QSt,at;θt∇Qs,a;θ.


### 3.6. Online Fine-Tuning

This section focuses on promoting the sustainability of EvadeRL, which is vital because malware is always evolving and its behavior and construction are changing rapidly [39]. The PDF malware classifiers trained on the most recent malware samples may capture different feature distributions, causing the EvadeRL trained on prior malware samples to deteriorate. DroidEvolver [40] managed to address this challenge for Android malware classifiers by creating a model pool with a variety of online learning algorithms that can perform incremental learning over streaming data. Similarly, the paradigm of pretraining the models using large-scale data and then fine-tuning the models on target tasks with less training data has achieved remarkable success in computer vision missions [41]. Both the online learning algorithms and fine-tuning improve the computational efficiency of training with new samples, compared with training the model from scratch. Motivated by the aforementioned work, EvadeRL leverages online fine-tuning to alleviate the problem of constantly evolving PDF malware samples and thereby obtains sustainability.Figure4 depicts the EvadeRL training workflow. During the pretraining stage, the agent interacts with a PDF malware detector that has been trained on previous malware samples and stores experience into the replay buffer. The replay buffer is used for pretraining the agent’s decision network as well as serving as a static dataset for fine-tuning the network in stage 2. When the PDF malware detector adapts to newer malware samples, EvadeRL leverages the static dataset and a small amount of online interaction between the agent and evolved PDF detector to fine-tune the decision network. EvadeRL is able to learn effectively from experience collected offline and fine-tune with a limited amount of online interaction data in order to build resilience against the evolution of PDF malware and detectors.Figure 4
EvadeRL’s pretraining and fine-tuning workflow.


### 3.7. Training

The training process is demonstrated in Algorithm 1 using pseudocode. Exploration allows an agent to improve its current knowledge about each modification action and make more informed decisions in the future, while exploitation chooses the most valuable mutation action to get the biggest reward by exploiting the agent’s current action-value estimates. To conquer the exploration and exploitation dilemma, theε-greedy policy is used, where ε refers to the probability of choosing to explore. The value of ε decreases from 1 to 0.1 when the training progresses according to (7) where n is the current training step and N is the total number of training steps.(7)ε=1−0.9×nN.To achieve greater data efficiency and break data correlation, the agent stores past experiences and learns from them repeatedly instead of discarding experiences after one stochastic gradient descent. Prioritized experience replay [42] is used in EvadeRL, which weighs the samples so that transactions with high significance are drawn more frequently for training. TD error is recalculated after the replay to measure the significance of each transition according to (8). The priority is updated according to TD error, and valuable transitions will get higher priority and are more possible to be replayed. In the content of generating adversarial examples, those who succeed in evading detection should have higher priority to be replayed.(8)δXi=rXi+γQ∗st+1Xi,argmaxaQstXi,aXi;θ;θ−−QsXi,aXi;θ.As shown in Algorithm 1, a priority replay buffer of capacityN is initialized at the beginning. The original PDF file, represented as pdf0, is mapped to the memory at the start of each episode. For each step, a mutation action is executed to modify the PDF file from pdft to pdft+1; pdft+1 is further transformed into the state st+1 that the agent can observe. After each step, the current state, the action taken, the reward, and the next state are stored as a quadruple (st, at, rt, st+1) into the priority replay buffer. After B transitions are stored, minibatches of samples are drawn with priority from the pool of stored samples and the experience replay begins. During the training procedure, the agent progressively learns the optimal estimation of state-action values through interaction with the environment and update of the policy.Algorithm 1: Training algorithm.
(1)
Initialize replay bufferM to capacity N and replay start size B(2)
Initialize two identical networks, action-value networkQ and target-value network Q∗, with random weights θ and weights θ−=θ respectively(3)
forepisode=1 to Ddo(4)
Initialize memory mappingpdf0 of a PDF file selected from malicious files List L(5)
Initialize states0=ExtractFeaturepdf0(6)
fort=1 to Tdo(7)
with probability∈ using exploration and exploitation strategy to select a random action at or choose at=argmaxaQst,a;θ(8)
Modifypdft to pdft+1 by action at(9)
Transformst by action at to st+1 and observer reward rt(10)
Store transitionst,at,rt,st+1 with priority into M(11)
ifsizeoM>Bthen(12)
fori=1 to Kdo(13)
Sample transitionXi∼PXi=pxi/∑jpj (j represents each transition in M)(14)
Compute TD -errorδxi ,using equation (6)(15)
Updatepxi=δxi(16)
Set loss function using equation (4)(17)
end for(18)
Exert Adam optimizer to optimize parameter Q(19)
end if(20)
EveryU step reset θ−=θ(21)
if done then(22)
break(23)
end if(24)
end for(25)
if episode mod TEST_INTERVAL==0then(26)
Run Testing Algorithm and record evasion rateER(27)
StoreQ and Q∗ to a new model M(28)
end if(29)
end forAlgorithm 2 summarizes the online fine-tuning algorithm. The static dataset is obtained via prior transitions of offline training. The replay buffer is initialized with the static dataset, and both networks are initialized with pretrained weights. The pretrained agent starts to interact with evolved PDF malware detector and collects online transition into the replay buffer. The parameters are then updated through a similar approach in Algorithm 1. As the networks are fine-tuned, the static dataset is also updated with newer transitions.Algorithm 2: Online fine-tuning.
(1)
Static datasetD=s,a,r,s′j(2)
Initialize replay bufferM=D(3)
Initialize action-value networksQ and target-value network Q∗ with pre-trained weights θ(4)
forepisode=1 to D′do(5)
Interact with evolved PDF classifierC and collect online transition t(6)
M←M∪t(7)
Sample batchs,a,r,s′j:M(8)
Update parameterθ according to Algorithm 1(9)
end for(10)
Update static dataset ()


## 3.1. Model Structure

The general structure of RL consists of an agent, an environment, a state space, an action space, and the reward. An appropriate definition of these elements is a necessity when using RL to construct a model to generate adversarial examples. Figure2 shows the model architecture of EvadeRL. EvadeRL uses DDQN [22] as its learning algorithm to reduce overestimation by decomposing the max operation in the target into action selection and action evaluation. DDQN evaluates the greedy policy according to the decision network Q but uses the target network Q∗ to estimate its value. The agent chooses an action to modify the given PDF file based on the current knowledge, the feature extractor and detector in the environment determine the state and classification result of the modified file, and both the state and reward are then returned to the agent afterward. Through the continuous interaction of the agent and the environment, the experience gained each time-step is stored in the replay buffer for the update of Q. The agent’s ultimate purpose is to choose actions in a fashion that succeeds in generating adversarial examples and maximizes the reward.Figure 2
The architecture of EvadeRL.The environment is comprised of a feature extractor and an integrated PDF detector; the former is responsible for observing the sample state, and the latter is bound to determine the reward to any given PDF example by the classification result. The environment uses the reward and the next state as feedback to the agent. The agent chooses mostly random actions to mutate PDF files according to theε-greedy policy at the early episodes. As the training process progresses, the agent uses the observation state and the reward returned by the environment to compute the state-action value and update the parameters of Q. Thus, the action-choosing policy of the agent can be more targeted by always choosing the optimal action according to (3), where st is the current state, θ represents the parameters of the decision network, a is a set of actions, and at denotes the best action the agent should choose. The action space only comprises different kinds of content insertions into the PDF files which are guaranteed not to crash the malicious functions and the original formats of them. The details of the EvadeRL model are elaborated as follows:(3)at=argmaxQst,a;θ.


## 3.2. Environment

The PDF files are mapped to the memory and exist in the environment as byte sequences. The raw bytes, however, are not appropriate for the agent to observe. In order to represent the current state of the PDF sample more accurately, the byte sequence is transformed into the form of a feature vector. Any irrelevant features should be removed from the feature set for the agent to better observe the environment. To this end, the initial observation state is defined as the set of features that will be influenced by the actions. After each action is taken, the observation state will change according to (4), where cntai represents how many times action ai has been taken and λ is a constant, while stfi denotes the observation space of feature fi at time-step t. Following the equation above, the agent is sensitive to the changing states of files and the situation where a single action is overfrequently taken can be avoided.(4)stfi=st−1fi−λcntai.


## 3.3. Action Space

The action space is specified by two basic rules: every action of mutation is effective in deceiving the target classifier, while the modification must not damage the original function of files. In addition, designing an action space containing rich actions in the problem scope of RL may lead to a difficult training process. Thus, a relatively lean action space is defined asA = {META, IF, IB, IOAS, MID}. As is shown in Figure 3, the modification methods inject contents only between the cross-reference table and the trailer without removing or modifying the existing elements. The injected content is ignored by PDF readers but can be discovered by the PDFRate detector.Figure 3
Modify by injecting contents between the cross-reference table and the trailer.The following are detailed descriptions of the actions: (1) META: Modify the metadata in the PDF files, which is general information such as the document’s title, author, and creation dates. In particular, both uppercase and lowercase letters along with digits and separators are appended to the specific metadata, including author, creator, producer, subject, and title. (2) IF: Insert a new type of font to the PDF file, which defines glyphs for a particular character set. (3) IR: Insert rectangles that are used to describe locations on a page and bounding boxes for a variety of objects. A rectangle is written as an array of four numbers giving the coordinates of a pair of diagonally opposite corners. Thus, an array shaped like [llx,lly,urx,ury] is inserted to represent a rectangle. (4) IOAS: Insert objects and stream objects. An object starts with the keyword “obj” and ends with “endobj,” while the keywords for stream objects are “stream” and “endstream.” This action is performed by inserting keywords and random content. (5) MID: Modify the identifier of the PDF file. The value of the ID field is an array of two strings, each of which has 32 letters and digits. If the provided PDF file does not contain any IDs, two identifiers are inserted into the trailer dictionary. In other circumstances, IDs are changed at random.


## 3.4. Reward

The reward is a scalar feedback signal from the environment that indicates how well the agent performs on a specific step. Since our goal is to force misclassification of the target detector, the intuition is to use its classification outcomes as the reward. The target detector is integrated into EvadeRL as a black-box detector, which only returns true or false for the input PDF files. The reward is defined as either a positive number to indicate the agent has succeeded in forcing misclassification after several actions or zero if this goal is not achieved and a maximum of steps have been taken.


## 3.5. Agent

The agent uses the neural network to estimate the value of different actions in a certain state and follow the policy of always taking the most valuable action at each state. There are two neural networks, that is, the decision networkQ and target network Q∗, which share the same network structure but serve different purposes. To make the training procedure stable, the parameter θ of Q is updated every step to fit Q∗, while θ− of Q∗ is only updated every C step and is held fixed between individual updates [38].In the context of evading the PDF detector, the decision network takes the observation state as input and then produces the next mutation action the agent will take. As for the output layer of the network, it is set to action-index in the action setA. In EvadeRL, the output dimension equals 5, which refers to the total dimension of the action space. The network uses both Q and Q∗ to compute the loss function, which is defined in (5), in which θt and θt− are the parameters of Q and Q∗. The parameters of the Q network are then updated by backpropagation using gradient descent, which is shown in (6).(5)Lw=ΕRt+1+γQSt+1,argmaxaSt+1,a;θt,θt−︸Target−QSt,at;θt2,(6)θt+1=θt+αRt+1+γQSt+1,argmaxaSt+1,a;θt,θt−−QSt,at;θt∇Qs,a;θ.


## 3.6. Online Fine-Tuning

This section focuses on promoting the sustainability of EvadeRL, which is vital because malware is always evolving and its behavior and construction are changing rapidly [39]. The PDF malware classifiers trained on the most recent malware samples may capture different feature distributions, causing the EvadeRL trained on prior malware samples to deteriorate. DroidEvolver [40] managed to address this challenge for Android malware classifiers by creating a model pool with a variety of online learning algorithms that can perform incremental learning over streaming data. Similarly, the paradigm of pretraining the models using large-scale data and then fine-tuning the models on target tasks with less training data has achieved remarkable success in computer vision missions [41]. Both the online learning algorithms and fine-tuning improve the computational efficiency of training with new samples, compared with training the model from scratch. Motivated by the aforementioned work, EvadeRL leverages online fine-tuning to alleviate the problem of constantly evolving PDF malware samples and thereby obtains sustainability.Figure4 depicts the EvadeRL training workflow. During the pretraining stage, the agent interacts with a PDF malware detector that has been trained on previous malware samples and stores experience into the replay buffer. The replay buffer is used for pretraining the agent’s decision network as well as serving as a static dataset for fine-tuning the network in stage 2. When the PDF malware detector adapts to newer malware samples, EvadeRL leverages the static dataset and a small amount of online interaction between the agent and evolved PDF detector to fine-tune the decision network. EvadeRL is able to learn effectively from experience collected offline and fine-tune with a limited amount of online interaction data in order to build resilience against the evolution of PDF malware and detectors.Figure 4
EvadeRL’s pretraining and fine-tuning workflow.


## 3.7. Training

The training process is demonstrated in Algorithm 1 using pseudocode. Exploration allows an agent to improve its current knowledge about each modification action and make more informed decisions in the future, while exploitation chooses the most valuable mutation action to get the biggest reward by exploiting the agent’s current action-value estimates. To conquer the exploration and exploitation dilemma, theε-greedy policy is used, where ε refers to the probability of choosing to explore. The value of ε decreases from 1 to 0.1 when the training progresses according to (7) where n is the current training step and N is the total number of training steps.(7)ε=1−0.9×nN.To achieve greater data efficiency and break data correlation, the agent stores past experiences and learns from them repeatedly instead of discarding experiences after one stochastic gradient descent. Prioritized experience replay [42] is used in EvadeRL, which weighs the samples so that transactions with high significance are drawn more frequently for training. TD error is recalculated after the replay to measure the significance of each transition according to (8). The priority is updated according to TD error, and valuable transitions will get higher priority and are more possible to be replayed. In the content of generating adversarial examples, those who succeed in evading detection should have higher priority to be replayed.(8)δXi=rXi+γQ∗st+1Xi,argmaxaQstXi,aXi;θ;θ−−QsXi,aXi;θ.As shown in Algorithm 1, a priority replay buffer of capacityN is initialized at the beginning. The original PDF file, represented as pdf0, is mapped to the memory at the start of each episode. For each step, a mutation action is executed to modify the PDF file from pdft to pdft+1; pdft+1 is further transformed into the state st+1 that the agent can observe. After each step, the current state, the action taken, the reward, and the next state are stored as a quadruple (st, at, rt, st+1) into the priority replay buffer. After B transitions are stored, minibatches of samples are drawn with priority from the pool of stored samples and the experience replay begins. During the training procedure, the agent progressively learns the optimal estimation of state-action values through interaction with the environment and update of the policy.Algorithm 1: Training algorithm.
(1)
Initialize replay bufferM to capacity N and replay start size B(2)
Initialize two identical networks, action-value networkQ and target-value network Q∗, with random weights θ and weights θ−=θ respectively(3)
forepisode=1 to Ddo(4)
Initialize memory mappingpdf0 of a PDF file selected from malicious files List L(5)
Initialize states0=ExtractFeaturepdf0(6)
fort=1 to Tdo(7)
with probability∈ using exploration and exploitation strategy to select a random action at or choose at=argmaxaQst,a;θ(8)
Modifypdft to pdft+1 by action at(9)
Transformst by action at to st+1 and observer reward rt(10)
Store transitionst,at,rt,st+1 with priority into M(11)
ifsizeoM>Bthen(12)
fori=1 to Kdo(13)
Sample transitionXi∼PXi=pxi/∑jpj (j represents each transition in M)(14)
Compute TD -errorδxi ,using equation (6)(15)
Updatepxi=δxi(16)
Set loss function using equation (4)(17)
end for(18)
Exert Adam optimizer to optimize parameter Q(19)
end if(20)
EveryU step reset θ−=θ(21)
if done then(22)
break(23)
end if(24)
end for(25)
if episode mod TEST_INTERVAL==0then(26)
Run Testing Algorithm and record evasion rateER(27)
StoreQ and Q∗ to a new model M(28)
end if(29)
end forAlgorithm 2 summarizes the online fine-tuning algorithm. The static dataset is obtained via prior transitions of offline training. The replay buffer is initialized with the static dataset, and both networks are initialized with pretrained weights. The pretrained agent starts to interact with evolved PDF malware detector and collects online transition into the replay buffer. The parameters are then updated through a similar approach in Algorithm 1. As the networks are fine-tuned, the static dataset is also updated with newer transitions.Algorithm 2: Online fine-tuning.
(1)
Static datasetD=s,a,r,s′j(2)
Initialize replay bufferM=D(3)
Initialize action-value networksQ and target-value network Q∗ with pre-trained weights θ(4)
forepisode=1 to D′do(5)
Interact with evolved PDF classifierC and collect online transition t(6)
M←M∪t(7)
Sample batchs,a,r,s′j:M(8)
Update parameterθ according to Algorithm 1(9)
end for(10)
Update static dataset ()


## 4. Experiments and Results

### 4.1. Datasets and Experiment Setup

The experiments were conducted on the Contagio [28] dataset which consists of 9000 benign and 10982 malicious PDF files and 11241 malware samples from VirusTotal [43] which were collected between 2017 and 2021. A total of 5000 benign files and 5000 malicious ones were used to extract 135 features and train the target classifier PDFRate. 5000 malicious PDF files were selected to train the EvadeRL model. 500 malicious files that have not been used for training were randomly selected for the test dataset, which was used to generate adversarial examples and validate the trained model.To set up the EvadeRL model, the environment is defined using the Gym [44] toolset, which is a toolkit provided by OpenAI for developing and comparing reinforcement learning algorithms. The basic class of environment named MalPDFEnv is extended to fit into the context of evading detection. With the definitions of observation space, action space, and two essential methods, reset and step, MalPDFEnv is set up and can be interacted with. Meanwhile, the agent in EvadeRL is implemented using ChainerRL [45] deep reinforcement learning library. The Cuckoo Sandbox [46], which can trace API calls and general file behavior and distill it into high-level information and signatures, is used to verify maliciousness after modification.For training EvadeRL, an optimizer of Adam algorithm with minibatch of size 32 is used, discount-factorγ uses 0.99, the replay buffer capacity N is set to 5000, and the replay buffer start-size B is set to 1000. The model can be trained for up to D = 10000 steps and T = 30 steps can be taken each episode; every 500 steps, the model is tested to evaluate its performance. The update interval of the target network is set to 100 steps, and it uses the “hard” update method.


### 4.2. Results

In this section, we have implemented four experiments to evaluate the performance of the proposed framework and answer the following research questions:(i)
Research Question A: How does the policy of the agent improve during training?(ii)
Research Question B: What is the distribution of classification scores of adversarial examples generated by EvadeRL?(iii)
Research Question C: How robust is EvadeRL against hardened detectors?(iv)
Research Question D: How much improvement can EvadeRL make in terms of evasion speed?(v)
Research Question E: Can EvadeRL maintain resilience against evolving PDF malware and detectors?


### 4.3. Experiment A: Performance Analysis for Policy Improvement through Training

To demonstrate that our approach has robustly learned a successful action-choosing policy during training, the averageQ value and the cumulative reward have been recorded to evaluate the improvement of policy. However, recording the cumulative reward is pointless in this work because the reward is either ten or zero, and it is only returned at the end of each episode. The average Q value is a measure of how well the current decision network predicts the value of actions and how much reward the agent can obtain following its current action-choosing policy.Figure5 depicts how the average Q value improved during training. As can be seen from the figure, the average Q is initially close to zero and increases slowly due to a lack of prior experience. However, it begins to soar up at around the 140th episode, and the curve is turned to be steady after the 300th episode. Despite the slight jittering, the curve suggests that the agent’s policy has reached optimum after training for around 400 episodes. The extensively improved policy of the agent allows it to accurately predict the value of actions and it can make wise actions to obtain the reward.Figure 5
The average value of action-value functionQ during the training process.Further analysis was carried out by comparing EvadeRL model (DDQN agent) with existing adversarial examples generating techniques based on DRL, including DQN based [47] and deep SARSA based [48] approaches. The reduction in the average number of steps required to generate adversarial examples is a direct manifestation of the optimization of the agent’s policy. Figure 6 illustrates the average number of steps taken per episode by three agents in each test round. The test rounds were conducted every 500 steps, each containing 50 testing episodes, and the upper bound of total steps for each episode is set to 30.Figure 6
Average steps taken per episode by three agents during test rounds.In the case of the DDQN agent, the results show that almost 28 steps have been taken per episode in the first test round, indicating that evasions are unlikely to succeed. The result of the second test round reveals that the training procedure is not as stable as expected. The subsequent test rounds have witnessed a steady reduction in the average steps per episode. After six rounds of training, the number of actions taken has decreased considerably. In the last three rounds shown in the figure, the average steps per episode remain consistent at around 14. The steps required to generate adversarial examples have decreased approximately 50% from the beginning to the ninth test round, which is strong evidence that the agent’s policy has evolved during training. In the case of the DQN agent, the average steps required to generate adversarial examples are more than the DDQN agent during all the testing rounds. However, the average steps of deep SARSA are fewer than those of DDQN during the first six testing rounds, which implies that deep SARSA learning converges faster than DDQN in the early stage of training. At the end of testing rounds, the average steps remain consistent at 14 for DDQN, 15 for deep SARSA, and 17 for DQN, indicating that DDQN learns better policy than others.As can be seen from Figure7, the evasion rate of the DDQN agent is initially 34% in the first testing round while reaching 100% in the eighth testing round. This figure reveals a similar conclusion that the deep SARSA agent generally achieves better evasion rates in the first five test rounds, while DDQN outperforms others in terms of the evasion rate at the end of training.Figure 7
Evasion rates of three agents during test rounds.


### 4.4. Experiment B: Classification Scores Distribution of Adversarial Examples

The investigation of classification scores distribution is essential to determine how successfully an evasion approach can deceive the detector and its applicability to various PDF samples. A box plot diagram has been drawn to demonstrate the comparison between our results and existing methods, including gray-box attack Mimicry [16] and fvGAN [21], and two black-box approaches, EvadeML [18] and EvadeHC [19]. Mimicry was operated by making the malicious file mimic different benign files 30 times to get the best scores. To implement fvGAN model, a GAN was trained to translate a malicious feature vector into an adversarial feature vector, which was further transformed into actual files. As gray-box attack requires extra knowledge of target classifiers, we reproduced Mimicry and fvGAN in two evasion scenarios, that is, FC and FTC. The labels F, T, and C denote prior knowledge of feature set, training dataset, and classification algorithm [16], respectively. The other three approaches were operated in black-box scenarios, where no knowledge was known in advance.In the box plot shown in Figure8, the dotted line indicates the median, and the upper and the lower edges of the square box denote the first and third quartiles of the scores, respectively. Besides, the top and bottom lines mean the maximum and minimum values of the result scores, while the extra dots are outliers. As can be seen from the figure, the classification scores of adversarial examples generated by EvadeRL are generally lower than others. In the scenario FC, the scores of Mimicry are distributed mostly between 0.16 and 0.51, whereas those of fvGAN are distributed between 0.23 and 0.49. In the scenario FTC, the fvGAN’s scores appear to be more centralized around 0.39. The score distributions of EvadeML and EvadeHC are similar, and the scores range mainly from 0.3 to 0.4. The scores of EvadeRL are distributed between 0.15 and 0.32, with a few outliers whose scores are higher than 0.4 or lower than 0.1. However, there are more evasive examples generated by Mimicry which score less than 0.1, indicating that they are close to benign files. Note that both Mimicry and fvGAN assume that adversaries have a deeper knowledge of the target detector and the feature space, which may allow them to modify the feature more precisely and lead to lower classification scores. In general, EvadeRL achieves greater evasive performance than others while obtaining less information of the target detector.Figure 8
Random Forest scores of adversarial examples generated by different approaches.


### 4.5. Experiment C: Effectiveness Evaluation on Hardened Detectors

An evaluation is necessary to ensure the robustness of EvadeRL against hardened classifiers; the first experiment was conducted by further lowering the PDFRate malware threshold. As illustrated in the previous work [27], different malware thresholds have little impact on PDFRate’s accuracy. By lowering the malware thresholds, it is much more difficult to generate adversarial examples. Thus, it can be used to simulate the scenario where the detectors are strengthened and evaluate the efficacy of EvadeRL and other former evasion techniques [16, 18, 19]. To reproduce EvadeML, the parameters were set the same as those mentioned in the paper [18]: the population size is 48, the maximum generation is 20, the mutation rate is 0.1, and the fitness stop threshold is 0.0.Table2 shows the evasion rates of Mimicry, EvadeML, EvadeHC, fvGAN, and EvadeRL against detectors with different malware thresholds. When the threshold was set to 0.45, both EvadeHC and EvadeML achieved a 100% evasion rate, while EvadeRL attained a 99% evasion rate, followed by 97% and 94% for fvGAN and Mimicry, respectively. As the threshold was lowered to 0.4, all the methods except EvadeRL suffered significant declines in evasion rates. When the threshold was further lowered to 0.35, EvadeRL still maintained an 86% evasion rate, while other approaches witnessed more significant drops in their evasion rates. The evasion rates dropped to 35% for Mimicry, 29% for EvadeML, 36% for EvadeHC, and 56% for fvGAN. In general, when the malware threshold is set to 0.45 or higher, the evasion rate of EvadeRL remains approximately the same as those of EvadeML and EvadeHC. With the malware threshold lowered to 0.4 or lower, EvadeRL is considerably more robust against PDFrate detection and surpasses other techniques in tougher scenarios, demonstrating that the policy learned by deep reinforcement learning is substantially more applicable in generating adversarial examples for a wider range of PDF samples.Table 2
Evasion rates of five approaches against hardened detectors with different malware thresholds.
MethodsThresholds0.45 (%)0.4 (%)0.35 (%)Mimicry947235EvadeML1005229EvadeHC1006736fvGAN977856EvadeRL999086Moreover, EvadeRL was evaluated on several state-of-the-art detectors to show the evasion effectiveness, including the image visualization detection technique (VisualPDF) [29] which claims to be robust and an extension of Hidost detector based on active learning (HidostAL) [30]. To set up VisualPDF, the PDF samples were preprocessed to byte plot using visualization techniques and texture features were extracted from generated images. VisualPDF was then trained using Random Forest algorithm. Table 3 illustrates the evasion rates of five approaches against different PDF malware detectors, and EvadeRL outperforms other approaches against all three detectors, while Mimicry attack is generally less effective against different PDF malware detectors. According to the results, VisualPDF showed the best robustness against adversarial examples by lowering the evasion rates to 41%, 82%, 88%, 67%, and 91%.Table 3
Evasion rates of five approaches against different PDF malware detectors.
MethodsDetectorsPDFRate (%)VisualPDF (%)HidostAL (%)Mimicry984156EvadeML10082100EvadeHC10088100fvGAN1006794EvadeRL10091100


### 4.6. Experiment D: Performance Analysis for Evasion Speed

To prove that EvadeRL is less computationally demanding due to fewer random mutations, the total evasion speed for generating 500 evasive examples by different approaches is shown in Figure9. The total evasion time includes parsing the original files, feature extraction, training the model, generating adversarial examples using the model, and validation of maliciousness of the generated files with the Cuckoo sandbox. The time required by EvadeML to generate 500 evasive variants against PDFRate is over 130 hours, while our approach achieves the same goal under the same experimental conditions within approximately 5 hours, which is more than 25 times faster. The stage of continuously interacting with the cuckoo sandbox to verify the maliciousness takes up a large portion of time spent by EvadeML and EvadeHC. Thus, the significant decrease of evasion time can be attributed to performing only maliciousness-preserving mutations so that the verification is required only once for each generated adversarial example.Figure 9
Total evasion time for generating 500 adversarial examples.


### 4.7. Experiment E: Sustainability Evaluation

The experiment was set up by first training the integrated detector and RL model on malware samples collected in 2011 and then retraining the detector with testing sets from 2017 to 2021, respectively. To evaluate the sustainability of EvadeRL with online fine-tuning, both the RL models with and without fine-tuning were set up to confront evolved detectors. Figure10 shows that the overtime performance of fine-tuned EvadeRL surpasses the regular RL model significantly. When evaluated on testing sets that are newer than training sets by six to ten years, the evasion rates of fine-tuned EvadeRL only decline by 3% at most, while those of regular model decline by 13.4% in the same case.Figure 10
Sustainability evaluation of EvadeRL with online fine-tuning.


## 4.1. Datasets and Experiment Setup

The experiments were conducted on the Contagio [28] dataset which consists of 9000 benign and 10982 malicious PDF files and 11241 malware samples from VirusTotal [43] which were collected between 2017 and 2021. A total of 5000 benign files and 5000 malicious ones were used to extract 135 features and train the target classifier PDFRate. 5000 malicious PDF files were selected to train the EvadeRL model. 500 malicious files that have not been used for training were randomly selected for the test dataset, which was used to generate adversarial examples and validate the trained model.To set up the EvadeRL model, the environment is defined using the Gym [44] toolset, which is a toolkit provided by OpenAI for developing and comparing reinforcement learning algorithms. The basic class of environment named MalPDFEnv is extended to fit into the context of evading detection. With the definitions of observation space, action space, and two essential methods, reset and step, MalPDFEnv is set up and can be interacted with. Meanwhile, the agent in EvadeRL is implemented using ChainerRL [45] deep reinforcement learning library. The Cuckoo Sandbox [46], which can trace API calls and general file behavior and distill it into high-level information and signatures, is used to verify maliciousness after modification.For training EvadeRL, an optimizer of Adam algorithm with minibatch of size 32 is used, discount-factorγ uses 0.99, the replay buffer capacity N is set to 5000, and the replay buffer start-size B is set to 1000. The model can be trained for up to D = 10000 steps and T = 30 steps can be taken each episode; every 500 steps, the model is tested to evaluate its performance. The update interval of the target network is set to 100 steps, and it uses the “hard” update method.


## 4.2. Results

In this section, we have implemented four experiments to evaluate the performance of the proposed framework and answer the following research questions:(i)
Research Question A: How does the policy of the agent improve during training?(ii)
Research Question B: What is the distribution of classification scores of adversarial examples generated by EvadeRL?(iii)
Research Question C: How robust is EvadeRL against hardened detectors?(iv)
Research Question D: How much improvement can EvadeRL make in terms of evasion speed?(v)
Research Question E: Can EvadeRL maintain resilience against evolving PDF malware and detectors?


## 4.3. Experiment A: Performance Analysis for Policy Improvement through Training

To demonstrate that our approach has robustly learned a successful action-choosing policy during training, the averageQ value and the cumulative reward have been recorded to evaluate the improvement of policy. However, recording the cumulative reward is pointless in this work because the reward is either ten or zero, and it is only returned at the end of each episode. The average Q value is a measure of how well the current decision network predicts the value of actions and how much reward the agent can obtain following its current action-choosing policy.Figure5 depicts how the average Q value improved during training. As can be seen from the figure, the average Q is initially close to zero and increases slowly due to a lack of prior experience. However, it begins to soar up at around the 140th episode, and the curve is turned to be steady after the 300th episode. Despite the slight jittering, the curve suggests that the agent’s policy has reached optimum after training for around 400 episodes. The extensively improved policy of the agent allows it to accurately predict the value of actions and it can make wise actions to obtain the reward.Figure 5
The average value of action-value functionQ during the training process.Further analysis was carried out by comparing EvadeRL model (DDQN agent) with existing adversarial examples generating techniques based on DRL, including DQN based [47] and deep SARSA based [48] approaches. The reduction in the average number of steps required to generate adversarial examples is a direct manifestation of the optimization of the agent’s policy. Figure 6 illustrates the average number of steps taken per episode by three agents in each test round. The test rounds were conducted every 500 steps, each containing 50 testing episodes, and the upper bound of total steps for each episode is set to 30.Figure 6
Average steps taken per episode by three agents during test rounds.In the case of the DDQN agent, the results show that almost 28 steps have been taken per episode in the first test round, indicating that evasions are unlikely to succeed. The result of the second test round reveals that the training procedure is not as stable as expected. The subsequent test rounds have witnessed a steady reduction in the average steps per episode. After six rounds of training, the number of actions taken has decreased considerably. In the last three rounds shown in the figure, the average steps per episode remain consistent at around 14. The steps required to generate adversarial examples have decreased approximately 50% from the beginning to the ninth test round, which is strong evidence that the agent’s policy has evolved during training. In the case of the DQN agent, the average steps required to generate adversarial examples are more than the DDQN agent during all the testing rounds. However, the average steps of deep SARSA are fewer than those of DDQN during the first six testing rounds, which implies that deep SARSA learning converges faster than DDQN in the early stage of training. At the end of testing rounds, the average steps remain consistent at 14 for DDQN, 15 for deep SARSA, and 17 for DQN, indicating that DDQN learns better policy than others.As can be seen from Figure7, the evasion rate of the DDQN agent is initially 34% in the first testing round while reaching 100% in the eighth testing round. This figure reveals a similar conclusion that the deep SARSA agent generally achieves better evasion rates in the first five test rounds, while DDQN outperforms others in terms of the evasion rate at the end of training.Figure 7
Evasion rates of three agents during test rounds.


## 4.4. Experiment B: Classification Scores Distribution of Adversarial Examples

The investigation of classification scores distribution is essential to determine how successfully an evasion approach can deceive the detector and its applicability to various PDF samples. A box plot diagram has been drawn to demonstrate the comparison between our results and existing methods, including gray-box attack Mimicry [16] and fvGAN [21], and two black-box approaches, EvadeML [18] and EvadeHC [19]. Mimicry was operated by making the malicious file mimic different benign files 30 times to get the best scores. To implement fvGAN model, a GAN was trained to translate a malicious feature vector into an adversarial feature vector, which was further transformed into actual files. As gray-box attack requires extra knowledge of target classifiers, we reproduced Mimicry and fvGAN in two evasion scenarios, that is, FC and FTC. The labels F, T, and C denote prior knowledge of feature set, training dataset, and classification algorithm [16], respectively. The other three approaches were operated in black-box scenarios, where no knowledge was known in advance.In the box plot shown in Figure8, the dotted line indicates the median, and the upper and the lower edges of the square box denote the first and third quartiles of the scores, respectively. Besides, the top and bottom lines mean the maximum and minimum values of the result scores, while the extra dots are outliers. As can be seen from the figure, the classification scores of adversarial examples generated by EvadeRL are generally lower than others. In the scenario FC, the scores of Mimicry are distributed mostly between 0.16 and 0.51, whereas those of fvGAN are distributed between 0.23 and 0.49. In the scenario FTC, the fvGAN’s scores appear to be more centralized around 0.39. The score distributions of EvadeML and EvadeHC are similar, and the scores range mainly from 0.3 to 0.4. The scores of EvadeRL are distributed between 0.15 and 0.32, with a few outliers whose scores are higher than 0.4 or lower than 0.1. However, there are more evasive examples generated by Mimicry which score less than 0.1, indicating that they are close to benign files. Note that both Mimicry and fvGAN assume that adversaries have a deeper knowledge of the target detector and the feature space, which may allow them to modify the feature more precisely and lead to lower classification scores. In general, EvadeRL achieves greater evasive performance than others while obtaining less information of the target detector.Figure 8
Random Forest scores of adversarial examples generated by different approaches.


## 4.5. Experiment C: Effectiveness Evaluation on Hardened Detectors

An evaluation is necessary to ensure the robustness of EvadeRL against hardened classifiers; the first experiment was conducted by further lowering the PDFRate malware threshold. As illustrated in the previous work [27], different malware thresholds have little impact on PDFRate’s accuracy. By lowering the malware thresholds, it is much more difficult to generate adversarial examples. Thus, it can be used to simulate the scenario where the detectors are strengthened and evaluate the efficacy of EvadeRL and other former evasion techniques [16, 18, 19]. To reproduce EvadeML, the parameters were set the same as those mentioned in the paper [18]: the population size is 48, the maximum generation is 20, the mutation rate is 0.1, and the fitness stop threshold is 0.0.Table2 shows the evasion rates of Mimicry, EvadeML, EvadeHC, fvGAN, and EvadeRL against detectors with different malware thresholds. When the threshold was set to 0.45, both EvadeHC and EvadeML achieved a 100% evasion rate, while EvadeRL attained a 99% evasion rate, followed by 97% and 94% for fvGAN and Mimicry, respectively. As the threshold was lowered to 0.4, all the methods except EvadeRL suffered significant declines in evasion rates. When the threshold was further lowered to 0.35, EvadeRL still maintained an 86% evasion rate, while other approaches witnessed more significant drops in their evasion rates. The evasion rates dropped to 35% for Mimicry, 29% for EvadeML, 36% for EvadeHC, and 56% for fvGAN. In general, when the malware threshold is set to 0.45 or higher, the evasion rate of EvadeRL remains approximately the same as those of EvadeML and EvadeHC. With the malware threshold lowered to 0.4 or lower, EvadeRL is considerably more robust against PDFrate detection and surpasses other techniques in tougher scenarios, demonstrating that the policy learned by deep reinforcement learning is substantially more applicable in generating adversarial examples for a wider range of PDF samples.Table 2
Evasion rates of five approaches against hardened detectors with different malware thresholds.
MethodsThresholds0.45 (%)0.4 (%)0.35 (%)Mimicry947235EvadeML1005229EvadeHC1006736fvGAN977856EvadeRL999086Moreover, EvadeRL was evaluated on several state-of-the-art detectors to show the evasion effectiveness, including the image visualization detection technique (VisualPDF) [29] which claims to be robust and an extension of Hidost detector based on active learning (HidostAL) [30]. To set up VisualPDF, the PDF samples were preprocessed to byte plot using visualization techniques and texture features were extracted from generated images. VisualPDF was then trained using Random Forest algorithm. Table 3 illustrates the evasion rates of five approaches against different PDF malware detectors, and EvadeRL outperforms other approaches against all three detectors, while Mimicry attack is generally less effective against different PDF malware detectors. According to the results, VisualPDF showed the best robustness against adversarial examples by lowering the evasion rates to 41%, 82%, 88%, 67%, and 91%.Table 3
Evasion rates of five approaches against different PDF malware detectors.
MethodsDetectorsPDFRate (%)VisualPDF (%)HidostAL (%)Mimicry984156EvadeML10082100EvadeHC10088100fvGAN1006794EvadeRL10091100


## 4.6. Experiment D: Performance Analysis for Evasion Speed

To prove that EvadeRL is less computationally demanding due to fewer random mutations, the total evasion speed for generating 500 evasive examples by different approaches is shown in Figure9. The total evasion time includes parsing the original files, feature extraction, training the model, generating adversarial examples using the model, and validation of maliciousness of the generated files with the Cuckoo sandbox. The time required by EvadeML to generate 500 evasive variants against PDFRate is over 130 hours, while our approach achieves the same goal under the same experimental conditions within approximately 5 hours, which is more than 25 times faster. The stage of continuously interacting with the cuckoo sandbox to verify the maliciousness takes up a large portion of time spent by EvadeML and EvadeHC. Thus, the significant decrease of evasion time can be attributed to performing only maliciousness-preserving mutations so that the verification is required only once for each generated adversarial example.Figure 9
Total evasion time for generating 500 adversarial examples.


## 4.7. Experiment E: Sustainability Evaluation

The experiment was set up by first training the integrated detector and RL model on malware samples collected in 2011 and then retraining the detector with testing sets from 2017 to 2021, respectively. To evaluate the sustainability of EvadeRL with online fine-tuning, both the RL models with and without fine-tuning were set up to confront evolved detectors. Figure10 shows that the overtime performance of fine-tuned EvadeRL surpasses the regular RL model significantly. When evaluated on testing sets that are newer than training sets by six to ten years, the evasion rates of fine-tuned EvadeRL only decline by 3% at most, while those of regular model decline by 13.4% in the same case.Figure 10
Sustainability evaluation of EvadeRL with online fine-tuning.


## 5. Conclusions

This paper has presented EvadeRL, a novel framework for crafting adversarial examples against the PDF detector based on DRL. The agent in EvadeRL model can choose optimal sequences of actions to generate adversarial examples which preserve the original maliciousness. Our approach is based on the black-box threat model, in which the adversaries have little knowledge other than the classification outcomes. Offline pretraining and online fine-tuning are adopted by EvadeRL to achieve sustainability against constantly evolving malware and detectors. Experiments were conducted to evaluate the effectiveness of EvadeRL against PDF malware detectors and compared the performance with those of the Mimicry, EvadeML, EvadeHC, and fvGAN in terms of the score distribution, evasion rates against different detectors, and execution cost. The results demonstrate that our approach can be substantially more time-efficient and robust while achieving approximately the same evasion rate as the state-of-the-art methods. However, this work has only concentrated on generating adversarial examples against the PDF detector. For future study, applying EvadeRL to other problem domains such as APK malware and further enhancing the model’s sustainability without constantly fine-tuning can be taken into consideration.



---

*Source: 7218800-2022-04-29.xml*