# A Dynamic Malicious Document Detection Method Based on Multi-Memory Features

Yuanyuan Wang, Gengwang Li, Min Yu, Kam-Pui Chow, Jianguo Jiang, Xiang Meng, Weiqing Huang

► To cite this version:

Yuanyuan Wang, Gengwang Li, Min Yu, Kam-Pui Chow, Jianguo Jiang, et al.. A Dynamic Malicious Document Detection Method Based on Multi-Memory Features. 19th IFIP International Conference on Digital Forensics (DigitalForensics), Jan 2023, Arlington, VA, United States. pp.203-218, 10.1007/978-3-031-42991-0_11. hal-05517477

HAL Id: hal-05517477

https://inria.hal.science/hal-05517477v1

Submitted on 18 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# A Dynamic Malicious Document Detection Method Based on Multi-Memory Features

Yuanyuan Wang<fnref n="1" />, Gengwang Li<fnref n="1" />, Min Yu<fnref n="1" />, Kam-Pui Chow<fnref n="2" />, Jianguo Jiang<fnref n="1" />, Xiang Meng<fnref n="1" />, and Weiqing Huang<fnref n="1" />

**Abstract.** The massive use of Microsoft Office documents underscores the need for effective malicious document detection techniques. Most detection methods characterize document behavior using application programming interface traces or other descriptive information, but ignore memory information due to inherent difficulties. Since many malicious behavior patterns are only manifested in memory, these detection methods are vulnerable to ubiquity evasion attacks. One difficulty in extracting malicious behavior information from memory is that only high-coverage memory dump sequences are meaningful, but no established methods can be employed. Another difficulty is that no efficient method exists for representing the numerous long memory dump sequences associated with malicious document samples.

This chapter describes a multi-memory-feature-based method that leverages memory information to detect malicious documents. The detection method employs a high-coverage memory dump service and a multiple memory dump sequence reduction approach. The memory dump service hooks system application programming interfaces to cover the entire lifetimes of processes while also monitoring the initial Office process and every spawned subprocess. The multiple memory dump sequence reduction approach efficiently represents each memory dump in terms of the difference from its adjacent dump. Ablation experiments demonstrate that the memory dump sequence reduction approach performs best using a long short-term memory classifier, yielding an accuracy of 98.27%. Experiments also demonstrate that the detection method outperforms state-of-the-art methods based on application programming interfaces in terms of accuracy and precision.

**Keywords:** Malicious Document Detection · Microsoft Office · Application Programming Interfaces · Dynamic Memory Analysis · Neural Networks

## 1 Introduction

The massive use of Microsoft Office documents worldwide makes them attractive targets for attacks that exploit various vulnerabilities to inject malicious

<footnote n="1">¹ Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China yumin@iie.ac.cn</footnote>

<footnote n="2">² University of Hong Kong, Hong Kong, China</footnote>
code. Kaspersky North America [9] reported that, during the second quarter of 2022, Microsoft Office vulnerabilities accounted for 82% of all exploited vulnerabilities [9]. Not surprisingly, malicious Office documents account for a large proportion of all types of malware.

Considerable research has focused on detecting malicious documents. The methods employ static analysis or dynamic analysis. Static analysis methods usually extract statistical features from Office document byte sequences [12, 19, 23, 27], content [10, 11, 15] and document structures [2, 14, 24, 26]. Static methods have high efficiency, but they are easily circumvented by structural changes, code obfuscation and other countermeasures. Dynamic methods are designed to address the limitations of static methods. They typically open documents in a deployed sandbox and observe and record their malicious behavior at runtime. Most dynamic methods rely on recorded application programming interface (API) sequences or reconstructed API callgraphs [16, 29, 31] to profile the runtime behavior of malicious documents. However, many malicious document behaviors are only manifested in memory, especially when decrypting sensitive text or unpacking control instructions.

Researchers have begun to analyze memory dumps to enhance detection, but their methods are designed to detect malware, not malicious documents. Moreover, the methods lack generalization capabilities for two reasons. One reason is that the dumped memory does not cover the entire lifetimes of malware samples. For example, many methods only analyze a single dump, which greatly increases the likelihood that they will miss malicious behavior [1, 6, 17, 20]. Javaheri et al. [8] dump memory at opportune times to increase the probability of capturing the malicious behavior, but the approach does not cover spawned subprocesses. The other reason is that an efficient technique is not available for reducing multiple long memory dump sequences to a suitable format for training classifiers. Some methods take pictures of entire memory dumps [1, 6] whereas others employ two-pass analyses of memory dumps [18, 20]. However, these approaches are not computationally efficient for processing long sequences of memory dumps.

The proposed multi-memory-feature-based method addresses the two limitations of existing memory dump methods. This is accomplished by leveraging a high-coverage memory dump service and a multiple memory dump sequence reduction approach. The dump service, which is implemented as a dynamic-linked library, hooks system APIs to ensure that it is automatically loaded into the process space of every subsequent spawned subprocess. After it is loaded, the service dumps memory from process space at a preset interval. Thus, the service creates dump sequences that cover the entire lifetimes of all the involved process and their spawned subprocesses.

The memory dump reduction approach processes each set of memory dump sequences. First, it transforms each dump to a token set and computes the token set difference for each pair of chronologically-adjacent dumps. Next, the memory differences associated with a single malicious document sample are concatenated to create a single document. Following this, the memory dump documents corresponding to the samples are fed to classifiers for training and testing.
An ablation experiment conducted to evaluate the memory dump sequence reduction approach compared it against three other dump reduction strategies using three common classifiers; the highest accuracy of 99.27% was obtained using the memory dump sequence reduction approach in conjunction with a long short-term memory (LSTM) classifier. Experiments also demonstrate that the proposed detection method outperforms the best state-of-the-art API-trace-based method [31] in terms of accuracy and precision.

# 2 Related Work

This section discusses static and dynamic analysis methods for detecting malicious documents. Static analysis methods, which extract features from byte sequences, code content and document structure, do not involve malicious document execution. Dynamic analysis methods execute and monitor malicious documents in instrumented environments.

## 2.1 Static Analysis Methods

Static analysis methods are divided into three categories based on the malicious document features they employ, statistical features, content features and structural features.

**Statistical-Feature-Based Methods** These methods directly extract character-level information from documents. The methods typically extract n-gram data from documents and compute the similarity [27], distribution change [23] or entropy [12, 19] of the data as features.

**Content-Feature-Based Methods** These methods focus on code contained in documents. Examples include counting the legal token sets in document code [10], extracting specific keywords [11] and computing the semantic similarity between codes [15] for use as features.

**Structural-Feature-Based Methods** These methods extract structural information from documents and then leverage keywords with the highest frequency [14], quantitative similarities of specific structural paths [2, 24] or integrated or abstracted structural information [26] as features.

## 2.2 Dynamic Analysis Methods

Although static methods are useful for detecting malicious documents, they are easily evaded by adversarial tactics such as adding benign features [25] or reducing unnecessary malicious features [30]. Dynamic analysis methods address the limitations of static methods by executing and monitoring malicious documents in instrumented environments. Depending on the features they leverage, dynamic analysis methods are characterized as script-based, API-based and memory-data-based:
**Script-Based Methods** These methods locate, extract and execute code in documents to determine if they are malicious. For example, MD-Scan [28] extracts JavaScript and runs it in a simulation environment. ZOOZLE [5] extracts and analyzes features from an abstract syntax tree constructed from JavaScript through dynamic execution. BISSAM [22] employs dynamic binary tools to automatically identify embedded malicious code. MPScan [13] dynamically executes samples to extract JavaScript code after de-obfuscation. Cujo [21], JSand [4] and Luxor [3] are detection methods based on script code. However, these methods have two limitations. One is the difficulty of accurately locating and extracting code in documents. The other is that they typically target a specific language and cannot detect other types of attacks.

**API-Based Methods** These methods mainly monitor and record the operations performed by documents in virtual environments. Sandboxes such as CWSandbox [29] and Cuckoo Sandbox [7] are commonly used by API-based methods. With the help of a sandbox, Zhang et al. [31] extracted API names and their parameters as features and Mohammed et al. [16] extracted API sequences as features, which they used to train malicious document classifiers. However, these methods have some limitations. Since sandboxes can only track and record system APIs, malware can evade detection by imitating the behavior of non-malicious samples. Also, the extracted APIs only have time-order relationships and do not incorporate call-order relationships that are often used by experts to manually identify malicious document activity. Additionally, these methods cannot discern important malicious behaviors that only exist in memory, such as decrypting sensitive text and unpacking control instructions.

**Memory-Data-Based Methods** These methods are gaining popularity because they address the limitations of script-based and API-based methods. The memory forensics component of the malware detection framework developed by Rathnayaka et al. [20] employed Cuckoo Sandbox to dump the entire memory of a virtual machine, following which Volatility was used to extract information such as the malware source, registry and dynamic link library. Nissim et al. [17] collected 3,000 memory dumps and represented each dump as a vector with 200 minhash values to train a machine-learning-based classifier. Dai et al. [6] transformed memory dump files into grayscale images in the PNG format and extracted histograms of oriented gradient (HOG) features of the images to train an artificial neural network. Bozkir et al. [1] employed two image descriptors, global image (GIST) and HOG, to analyze the images created from memory dumps.

However, the above methods only dump memory once during a sample run, so there is no guarantee that malicious behavior has already occurred or is occurring at the time of dumping. Also, after the detection mechanism is understood, detection can be evaded by controlling the time of attack and packing or encrypting malicious code. Javaheri et al. [8] employed considerable empirical analysis
to determine the right time to dump memory, but only achieved a success rate of 53.88% for unpacking packed files.

To address this limitation, some methods rely on multiple memory dumps during a sample run. For example, Panker et al. [18] acquired 100 memory dumps from a running Linux server every ten seconds and used Volatility to extract 171 features. However, dumping memory multiple times to ensure coverage of malicious code execution is expensive in terms of space and processing time [18, 20]. Additionally, dumped memory cannot be used directly to train classifiers. In fact, current memory dump processing methods that represent dumps as pictures [1, 6] cannot handle multiple memory dumps at the same time.

Unlike the methods discussed in this section, the proposed malicious document detection method considers the entire execution of samples, including their subprocesses. Additionally, it efficiently handles multiple memory dumps while reducing noise.

# 3 Proposed Detection Method

This section describes the proposed multi-memory-feature-based malicious document detection method that relies on memory dumps. The implementation incorporates two modules, one implementing a high-coverage memory dump service and the other implementing an efficient algorithm for representing long memory dump sequences. The high coverage property is vital to a memory-dump-based method because the goal is to capture malicious document behavior to the maximal extent. The efficient representation algorithm is vital because it makes the utilization of the method possible.

## 3.1 Overview

Figure 1 shows the malicious document workflow. Given an Office document sample, Cuckoo Sandbox launches the Office document process in a separate sandbox environment. Meanwhile, the memory dump service is automatically loaded into process space for the initial Office document process as well as for all the processes in its control flow. Once it is loaded, each memory dump service periodically dumps the memory of its process. After all the processes have completed their execution, multiple sequences of memory dumps are generated for the document sample. Each dump sequence corresponds to a single process and the multiple sequences are sorted in chronological order.

The representation algorithm transforms the multiple memory dump sequences associated with a single document sample into a compact form for classifier model training. Specifically, each memory dump is tokenized to a word bag. Next, each word bag is replaced by the set distance between itself and its temporally-adjacent word bag. Following this, all the word bags are concatenated in chronological order into a compact raw representation of the document sample. Finally, the document sample representation is fed to pre-trained downstream classifiers that output the probability that the sample Office document is malicious.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Malicious document detection workflow.</figcaption></figure>

## 3.2 High-Coverage Dump Service

Existing memory-based dynamic detection methods create very few memory dumps during the execution of Office document processes; consequently, they have high probabilities of missing malicious document behaviors. Indeed, the efficacy of memory-based malicious document detection is enhanced by dumping memory multiple times to achieve high coverage. High coverage is measured in two dimensions. One is the time domain, meaning that the entire lifetime of every process is covered. The other is the spatial domain, meaning that every process starting with the initial Office document process and all the processes spawned by the control flow of the initial process are covered.

To achieve maximal coverage, the memory dump service is designed to have high coverage in the time and spatial domains. The memory dump service injects into each process a procedure that dumps its memory space periodically using a background thread.

**High Time Domain Coverage** The memory dump procedure is injected into each process space to dump its memory periodically until it terminates. To ensure that malicious behavior is not missed before the injection, each process is started in the pending mode and resumed only after the dump procedure is injected. Once the process is resumed, the dump procedure dumps memory at a preset interval. Each memory dump is labeled with the process PID and dumping timestamp. Thus, the dump procedure generates a memory dump sequence for each process sorted in chronological order.

The system API MiniDumpWriteDump was selected from among several dump utilities because it provides rich dump functionality with a flexible inter-
face and is accessible on any Windows platform. Additionally, the utility supports various dump types, each of which dumps different areas of process memory space. Experiments revealed that the MiniDumpNormal type dumps small amounts of memory while achieving good performance in downstream tasks. By invoking MiniDumpWriteDump with the argument MiniDumpNormal, only the information necessary to capture stack traces for all existing threads of a process was dumped.

**High Spatial Domain Coverage** The spatial domain refers to the set of processes spawned by the control flow of the initial Office document process. Since Office processes may launch subprocesses with malicious behavior, the dump service should cover them too. This is achieved by hooking CreateProcess, the system API used to create processes for Office executables. By hooking CreateProcess, a subprocess is created in the pending mode and the dump procedure is injected into the subprocess before it is resumed. The system APIs of created subprocesses are hooked similarly so that their spawned subprocesses are also injected with the dump procedure before they are resumed. Thus, the dump procedure is propagated along the control flow of the initial Office process with high coverage.

Another system API, CreateRemoteThread, which runs a piece of local code in remote process space, is also hooked. This enables the dump service to cover malicious behavior by the local code.

**Cuckoo Sandbox** The dump service is implemented using Cuckoo Sandbox, an open-source dynamic analysis platform. Cuckoo Sandbox comprises a host and clients connected via a virtual network. The core of Cuckoo Sandbox is its hooking mechanism, which is implemented by *monitor.dll* and *inject.exe* compiled by CucMonitor (Cuckoo Sandbox Monitor).

Figure 2 shows the Cuckoo Sandbox workflow. The host starts the virtual machine after receiving the analysis task submitted by the user and transmits the sample and analysis code to the client. The client executes the analysis code, starts the specified process to run the sample according to the sample type, calls *inject.exe* to suspend the new process and injects *monitor.dll* into it. The injected *monitor.dll* hooks the process to monitor its function calls and intercept and modify its execution process. After the analysis is completed, the client submits the analysis results to the host and restores itself to the original snapshot state.

The hooking mechanism of Cuckoo Sandbox is exactly what is needed by the proposed detection method, so it is leveraged to achieve high coverage of the spatial domain. The memory dump service is coded in CucMonitor and compiled into *monitor.dll*, enabling the memory dump service to be injected into process space within *monitor.dll*. In addition to Cuckoo Sandbox's original operation of API functions, attention is paid to calling the CreateProcess API. By hooking this API, it is possible to dynamically obtain parameters such as the IP address
<figure><img src="image_4.png" /><figcaption>Fig. 2. Cuckoo Sandbox workflow.</figcaption></figure>

and name of the process that are used to bind the process to the memory dump service.

**Memory Dump Sequence Reduction** The set of memory dump sequences for each malicious document sample must be transformed to a representation suitable for classifier training. As mentioned earlier, methods described in the literature only employ one memory dump for each malicious document sample. In contrast, in the proposed method, each malicious document sample yields a set of memory dump sequences where each sequence corresponds to one involved process. It is non-trivial to efficiently represent a malicious document sample in terms of such memory dump sequences.

Inspired by contrast learning and ResNet, the differences between chronologically-adjacent dumps are employed to create an efficient representation for classifier training. This notion is incorporated in the memory dump sequence reduction approach specified in Algorithm 1. The algorithm takes a set of memory dump sequences $S$ from the same malicious document sample as input. For each memory dump sequence $S_{pid}$ from the same process, the difference $D_t$ is computed for each pair of adjacent memory dumps, which are concatenated in chronological order to create a single document $R_{pid}$. All the concatenated documents $R_{pid}$ of all the spawned processes (subprocesses) are eventually merged in spawned order to create a single document $R$.

The dump difference $D_t$ is created by first tokenizing each memory dump $m_t$ into a meaningful token set. Next, the token frequencies are computed and stored in $Table_t$ for each memory dump $m_t$. Following this, the difference between two adjacent counting tables $Table_t$ and $Table_{prev}$ is computed as the memory dump
**Algorithm 1:** Memory dump sequence reduction.

**Data:** Sample $S = \{S_{pid} | pid$ is the process id\}$ where

$S_{pid} = \{m_t | t$ is the dump time\}$ is a memory dump sequence and
$m_t$ is the memory dump at time $t$.

**Result:** Sample Representation $R$

$R \leftarrow EmptyString$

**foreach** $S_{pid} \in S$ **do**

$R_{pid}, Table_{prev} \leftarrow EmptyString, EmptyTable$

**foreach** $m_t \in S_{pid}$ **do**

Tokens $\leftarrow TOKENIZE(m_t)$

Table$_t \leftarrow COUNT(Tokens)$

$D_t \leftarrow Table_t \setminus Table_{prev}$

$R_{pid} \leftarrow R_{pid} \oplus JOIN(D_t)$

Table$_{prev} \leftarrow Table_t$

**end**

$R \leftarrow R \oplus R_{pid}$

**end**

difference $D_t$. Finally, all the tokens in the difference tables are concatenated to
create the document $R$.

The effectiveness and efficiency of the memory dump sequence reduction
algorithm are assessed later in this chapter by comparing it with three other
memory dump reduction strategies:

- **RandMemDump Strategy:** This strategy randomly selects exactly one memory dump from the initial Office process for each sample and uses the memory dump as a document without any further processing. This strategy is similar to the one used in [1, 6, 17, 20] that only considers the memory state at a certain time.

- **AllMemDumps Strategy:** This strategy is like the RandMemDump strategy, except that it concatenates all the memory dumps of one sample into an oversized document. This strategy is the simplest way to represent long sequences of memory dumps. While the strategy effectively uses multiple memory dumps, the oversized documents it produces are much too large for classifier training.

- **ByteByByteDiff Strategy:** This strategy computes memory dump differences in a byte-by-byte manner. It employs the same representation created by Algorithm 1.

# 4 Experimental Evaluation and Results

This section describes the experimental dataset, experimental setup, evaluation
metrics, experiments and experimental results.
<table><caption>**Table 1.** Experimental dataset.</caption><thead><tr><th>Label</th><th>File Type</th><th>Number</th></tr></thead><tbody><tr><td rowspan="4">Malicious</td><td>DOC</td><td>1,2847</td></tr><tr><td>DOCX</td><td>2,483</td></tr><tr><td>XLS</td><td>5,082</td></tr><tr><td>XLSX</td><td>852</td></tr><tr><td>Total</td><td></td><td>21,264</td></tr><tr><td rowspan="4">Benign</td><td>DOC</td><td>3,302</td></tr><tr><td>DOCX</td><td>3,608</td></tr><tr><td>XLS</td><td>2,734</td></tr><tr><td>XLSX</td><td>3,164</td></tr><tr><td>Total</td><td></td><td>12,808</td></tr></tbody></table>

## 4.1 Experimental Dataset

The experimental dataset employed to evaluate the proposed multi-memory-feature-based malicious document detection method comprised 21,264 malicious samples and 12,808 benign samples. Table 1 shows details about the samples, which included DOC, DOCX, XLS and XLSX files. The malicious samples were drawn from Virus Share, a relatively authoritative malicious file sharing website that has published multiple malicious sample datasets. The benign document samples were collected using crawlers with Google, Bing, Baidu and other search engines.

The experiments employed subsets of the dataset, 12,500 malicious samples and 12,500 benign samples. The training, verification and testing sets were obtained by dividing the subsets in the ratio 6:2:2.

## 4.2 Experimental Setup

Sample analysis, memory extraction and memory preprocessing employed computers equipped with an Intel(R) Xeon (R) CPU E5-2620 v2 2.10 GHz and AS-PEED Graphics Family (rev. 21). Cuckoo Sandbox, which was deployed on a computer, controlled 12 Windows XP clients with the help of VirtualBox. After a sample was submitted, Cuckoo Sandbox performed sample analysis for a duration of one minute. The time interval for memory dumps was set to 500 s.

After analysis was completed, multiple memory dumps, grouped by the processes related to each sample, were transferred to the host. For each memory dump, the *strings* command was used to extract valid strings in memory. Subsequently, all the memory dumps of a single process were sorted by time and inserted in a set. Finally, Algorithm 1 was executed to extract the memory features. It was possible to decide whether or not to process the memory files of subprocesses. The processing of a subprocess was the same as that of its process and the extracted features were spliced after the process sequence.
<table><caption>**Table 2.** Performance comparison against an API-feature-based method.</caption><thead><tr><th>Method</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1-Score</th></tr></thead><tbody><tr><td>API-Feature-Based Method</td><td>0.9758</td><td>0.9849</td><td>0.9664</td><td>0.9756</td></tr><tr><td>Proposed Method</td><td><b>0.9874</b></td><td><b>0.9927</b></td><td><b>0.9820</b></td><td><b>0.9873</b></td></tr></tbody></table>

## 4.3 Evaluation Metrics

The accuracy, precision, recall and F1-score metrics were employed in the evaluation:

$$ \text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}} $$

$$ \text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}} $$

$$ \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}} $$

$$ \text{F1-Score} = 2 \times \text{Precision} \times \frac{\text{Recall}}{\text{Precision} + \text{Recall}} $$

where true positives TP is the number of malicious samples identified as malicious, false negatives TN is the number of benign samples identified as benign, false positives FP is the number of benign samples identified as malicious and false negatives FN is the number of malicious samples identified as benign.

## 4.4 Experiments and Results

Four sets of experiments were performed. This section describes the experiments and the experimental results.

**Detection Effectiveness** This set of experiments compared the performance of the proposed method against the performance of the traditional dynamic analysis method of Zhang et al. [31], which extracts API functions and their parameter information. The method is by far the most commonly used detection method with the richest feature extraction. To eliminate the impact of memory extraction on the original Cuckoo Sandbox analysis report, the samples were analyzed using a clean Cuckoo Sandbox for a duration of one minute. After obtaining the analysis report for a sample, the API and its parameters were extracted from the report as the feature representation of the sample. In the experiments, the sample features based on API functions and parameters and the sample features based on memory dump differences employed the same word vector representations and the same three network models for training.

Table 2 compares the performance of the proposed method against the performance of an API-feature-based method [31]. For each detection method, the
<table><caption>**Table 3.** Performance comparison of memory dump processing strategies.</caption><thead><tr><th>Method</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1-Score</th></tr></thead><tbody><tr><td>RandMemDump</td><td>0.9614</td><td>0.9788</td><td>0.9432</td><td>0.9607</td></tr><tr><td>AllMemDumps</td><td>0.9754</td><td>0.9873</td><td>0.9632</td><td>0.9751</td></tr><tr><td>ByteByByteDiff</td><td>0.9818</td><td>0.9934</td><td>0.970</td><td>0.9816</td></tr><tr><td>Proposed Method (Main Process)</td><td><strong>0.9888</strong></td><td><strong>0.9900</strong></td><td><strong>0.9876</strong></td><td><strong>0.9888</strong></td></tr><tr><td>Proposed Method</td><td><strong>0.9874</strong></td><td><strong>0.9927</strong></td><td><strong>0.9820</strong></td><td><strong>0.9873</strong></td></tr></tbody></table>

table presents the best performance metric results from among the three network models. The results demonstrate that the proposed method outperforms the API-feature-based method. This is because the API-feature-based method only records the system APIs called by the samples whereas the proposed method extracts all the operations performed by the samples by way of memory differences. Since the proposed method effectively incorporates more information, it has better accuracy. Additionally, the API-feature-based method leverages API parameter information and time relationships, which correspond to the dependencies between APIs. In the case of the proposed method, code statements dynamically change the memory stack by pushes and pops, and the dependencies between code statements become obvious.

**Feature Selection** This set of experiments compared the performance of the memory features used by the proposed method against the performance of three other memory features, RandMemDump, AllMemDumps and ByteByByteDiff. The performance of the proposed method for the main process and its subprocesses and the performance for the main process alone are also presented for comparison purposes.

Table 3 compares the performance of five memory dump processing strategies used in the experiments. The performance metrics of the proposed method with the main process and subprocesses and with the main process alone are superior to the performance metrics of the three other memory features.

RandMemDump uses a single memory dump and, therefore, incorporates too little effective information. AllMemDumps uses multiple memory dumps, but incorporates too much noisy information. Thus, RandMemDump and AllMemDumps do not represent the samples well and have lower performance metrics. The comparisons with RandMemDump and AllMemDumps demonstrate that the high-coverage multiple memory dumps of the proposed method incorporate more useful information that enhances detection.

ByteByByteDiff uses multiple memory dumps and has high accuracy but low recall, resulting in a lower F1-score than the proposed method. This is because the byte relationships in memory dumps were destroyed when valid strings were collected, leading to deviations in feature extraction by ByteByByteDiff and, ultimately, reduced accuracy. The comparisons with ByteByByteDiff and
<table><caption>**Table 4.** Performance comparison of deep learning models.</caption><thead><tr><th>Model</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1-Score</th></tr></thead><tbody><tr><td>Fasttext</td><td>0.9790</td><td>0.9914</td><td>0.9664</td><td>0.9787</td></tr><tr><td>LSTM</td><td><strong>0.9874</strong></td><td><strong>0.9927</strong></td><td><strong>0.9820</strong></td><td><strong>0.9873</strong></td></tr><tr><td>Transformer</td><td>0.9680</td><td>0.9899</td><td>0.9456</td><td>0.9673</td></tr></tbody></table>

AllMemDumps demonstrate the efficiency of multiple memory dump sequence reduction.

Curiously, the proposed method executed with only the main process is slightly superior to the proposed method with the main process and subprocesses for all the performance metrics except for recall. A possible reason is that the merging of subprocess differences may have resulted in the loss of subprocess timing relationships. Another possibility is that the dataset may not be rich enough because the subprocesses mostly executed legal operations instead of malicious operations.

**Model Selection** This set of experiments compared the performance of three commonly-used deep learning models, FastText, LSTM and Transformer, to determine the best model for learning memory features. FastText is a leading neural network model that greatly reduces training time while maintaining classification accuracy. An LSTM is a special recurrent neural network structure that learns long-term correlations and effectively processes samples comprising long sequences. A transformer employs a self-attention mechanism to connect an encoder and decoder with good parallelism.

Table 4 compares the performance of the three deep learning models. The LSTM model has the best performance across the four metrics, indicating that its training was the most effective. This is because each sample is represented in terms of the differences between multiple memory dumps with temporal relationships. The LSTM model is designed to learn long-term dependencies and effectively processes samples with temporal relationships. The model is best suited to handle the proposed memory features and makes good use of them.

**Processing Efficiency** The fourth set of experiments analyzed the time performance of four memory dump processing strategies, RandMemDump, AllMemDumps, ByteByByteDiff, and the proposed multiple memory dump reduction approach. The proposed multiple memory dump reduction approach was used with the LSTM model, the deep learning model with the best detection effectiveness and performance. In the case of the proposed approach, only the time required for partitioning memory dumps and memory feature extraction was recorded. Since memory collection and sandbox analysis occurred synchronously and were set to durations of one minute, memory collection did not affect the time performance.
<table><caption>**Table 5.** Time comparison of memory dump processing strategies.</caption><thead><tr><td>Feature</td><td>Time (s)</td></tr></thead><tbody><tr><td>RandMemDump</td><td>190.36</td></tr><tr><td>AllMemDumps</td><td>1,215.02</td></tr><tr><td>ByteByByteDiff</td><td>1,451.53</td></tr><tr><td>Proposed Method</td><td><b>609.10</b></td></tr></tbody></table>

Table 5 shows the times required by the four strategies for processing memory dumps and presenting samples. As expected, RandMemDump, which uses a single dump is the fastest but, as shown in Table 3, its detection performance is not good. Merging dumps in the AllMemDumps strategy requires considerable system resources, so the time requirement is high but, as shown in Table 3, the detection performance is not greatly improved. ByteByByteDiff is the slowest because it requires one more traversal compared with the other strategies. The multiple memory dump sequence reduction algorithm used in the proposed method requires less time than the two other multiple memory dump strategies, AllMemDumps and ByteByByteDiff, but, as shown in Table 3, has the best detection performance.

## 5 Conclusions

The multi-memory-feature-based malware detection method for Microsoft Office documents leverages a high-coverage memory dump service and a multiple memory dump sequence reduction approach. By hooking specific APIs, the memory dump service covers the entire lifetimes of the processes and subprocesses in the control flow of the initial Office process. This effectively extracts more information from memory that is leveraged to improve detection accuracy. The multiple memory dump sequence reduction approach efficiently represents memory dumps in terms of the differences from their chronologically-adjacent dumps. The approach concatenates all the differences between all the pairs of adjacent memory dumps into a compact, trainable format that embodies rich behavior information. The approach also can process multiple memory dumps at the same time while reducing noise and the training time.

Experiments demonstrate that the proposed malicious document detection method has an accuracy of 98.74%, which is 1.16% higher than the best traditional API-based detection method. Also, the method yields excellent results with an accuracy rate of at least 0.56% higher than other memory features with a time requirement of just 609.10 s.

## References

1. A. Bozkir, E. Tahillioglu, M. Aydos and I. Kara, Catch them alive: A malware detection approach through memory forensics, manifold learning and computer
vision, *Computers and Security*, vol. 103, article no. 102166, 2021.

2. A. Cohen, N. Nissim, L. Rokach and Y. Elovici, SFEM: Structural feature extraction methodology for the detection of malicious Office documents using machine learning methods, *Expert Systems with Applications*, vol. 63, pp. 324–343, 2016.

3. I. Corona, D. Maiorca, D. Ariu and G. Giacinto, LuxOR: Detection of malicious PDF-embedded JavaScript code through discriminant analysis of API references, *Proceedings of the Workshop on Artificial Intelligence and Security*, pp. 47–57, 2014.

4. M. Cova, C. Kruegel and G. Vigna, Detection and analysis of drive-by-download attacks and malicious JavaScript code, *Proceedings of the Nineteenth International Conference on the World Wide Web*, pp. 281–290, 2010.

5. C. Curtsinger, B. Livshits, B. Zorn and C. Seifert, ZOZZLE: Fast and precise in-browser JavaScript malware detection, *Proceedings of the Twentieth USENIX Security Symposium*, 2011.

6. Y. Dai, H. Li, Y. Qian, R. Yang and M. Zheng, SMASH: A malware detection method based on multi-feature ensemble learning, *IEEE Access*, vol. 7, pp. 112588–112597, 2019.

7. C. Guarnieri, M. Schloesser, J. Bremer and A. Tanasi, Cuckoo Sandbox open-source automated malware analysis, presented at *Black Hat USA*, 2013.

8. D. Javaheri and M. Hosseinzadeh, A framework for recognition and confronting of obfuscated malware based on memory dumping and filter drivers, *Wireless Personal Communications*, vol. 98(1), pp. 119–137, 2018.

9. Kaspersky North America, Eight times more users attacked via an old Microsoft Office vulnerability in Q2, Press Release, Woburn, Massachusetts (www.kaspersky.com/about/press-releases/2022_eight-times-more-users-attacked-via-an-old-microsoft-office-vulnerability-in-q2), August 15, 2022.

10. P. Laskov and N. Srndic, Static detection of malicious JavaScript-bearing PDF documents, *Proceedings of the Twenty-Seventh Annual Computer Security Applications Conference*, pp. 373–382, 2011.

11. J. Lin and H. Pao, Multi-view malicious document detection, *Proceedings of the Conference on Technologies and Applications of Artificial Intelligence*, pp. 170–175, 2013.

12. L. Liu, X. He, L. Liu, L. Qing, Y. Fang and J. Liu, Capturing the symptoms of malicious code in electronic documents by file entropy signals combined with machine learning, *Applied Soft Computing*, vol. 82, article no. 105598, 2019.

13. X. Lu, J. Zhuge, R. Wang, Y. Cao and Y. Chen, De-obfuscation and detection of malicious PDF files with high accuracy, *Proceedings of the Forty-Sixth Hawaii International Conference on System Sciences*, pp. 4890–4899, 2013.

14. D. Maiorca, G. Giacinto and I. Corona, A pattern recognition system for malicious PDF file detection, *Proceedings of the Eighth International Workshop on Machine Learning and Data Mining in Pattern Recognition*, pp. 510–524, 2012.

15. M. Mimura and T. Ohminami, Using LSI to detect unknown malicious VBA macros, *Journal of Information Processing*, vol. 28, pp. 493–501, 2020.

16. T. Mohammed, L. Nataraj, S. Chikkagoudar, S. Chandrasekaran and B. Manjunath, HAPSSA: Holistic approach to PDF malware detection using signal and statistical analysis, *Proceedings of the IEEE Military Communications Conference*, pp. 709–714, 2021.

17. N. Nissim, O. Lahav, A. Cohen, Y. Elovici and L. Rokach, Volatile memory analysis using the minhash method for efficient and secure detection of malware in private clouds, *Computers and Security*, vol. 87, article no. 101590, 2019.
18. T. Panker and N. Nissim, Leveraging malicious behavior traces from volatile memory using machine learning methods for trusted unknown malware detection in Linux cloud environments, *Knowledge-Based Systems*, vol. 226, article no. 107095, 2021.

19. H. Pareek, P. Eswari and N. Babu, Entropy and n-gram analysis of malicious PDF documents, *International Journal of Engineering and Technology*, vol. 2(2), 2013.

20. C. Rathnayaka and A. Jamdagni, An efficient approach for advanced malware analysis using a memory forensic technique, *Proceedings of the Sixteenth IEEE International Conference on Trust, Security and Privacy in Computing and Communications, Eleventh IEEE International Conference on Big Data Science and Engineering and Fourteenth IEEE International Conference on Embedded Software and Systems*, pp. 1145-1150, 2017.

21. K. Rieck, T. Krueger and A. Dewald, Cujo: Efficient detection and prevention of drive-by-download attacks, *Proceedings of the Twenty-Sixth Annual Computer Security Applications Conference*, pp. 31-39, 2010.

22. T. Schreck, S. Berger and J. Gobel, BISSAM: Automatic vulnerability identification of Office documents, *Proceedings of the International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment*, pp. 204-213, 2012.

23. M. Shafiq, S. Khayam, and M. Farooq, Embedded malware detection using Markov n-grams, *Proceedings of the Fifth International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment*, pp. 88-107, 2008.

24. N. Srndic and P. Laskov, Detection of malicious PDF files based on hierarchical document structure, *Proceedings of the Twentieth Annual Network and Distributed System Security Symposium*, 2013.

25. N. Srndic and P. Laskov, Practical evasion of a learning-based classifier: A case study, *Proceedings of the IEEE Symposium on Security and Privacy*, pp. 197-211, 2014.

26. N. Srndic and P. Laskov, Hidost: A static machine-learning-based detector of malicious files, *EURASIP Journal on Information Security*, vol. 2016(1), article no. 45, 2016.

27. S. Stolfo, K. Wang and W. Li, Towards stealthy malware detection, in *Malware Detection*, M. Christodorescu, S. Jha, D. Maughan, D. Song and C. Wang (Eds.), Springer, Boston, Massachusetts, pp. 231-249, 2007.

28. Z. Tzermias, G. Sykiotakis, M. Polychronakis and E. Markatos, Combining static and dynamic analysis for the detection of malicious documents, *Proceedings of the Fourth European Workshop on System Security*, article no. 4, 2011.

29. C. Willems, T. Holz and F. Freiling, Toward automated dynamic malware analysis using CWSandbox, *IEEE Security and Privacy*, vol. 5(2), pp. 32-39, 2007.

30. W. Xu, Y. Qi and D. Evans, Automatically evading classifiers: A case study on PDF malware classifiers, *Proceedings of the Twenty-Third Network and Distributed Systems Symposium*, vol. 10, 2016.

31. Z. Zhang, P. Qi and W. Wang, Dynamic malware analysis with feature engineering and feature learning, *Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, Thirty-Second Innovative Applications of Artificial Intelligence Conference and Tenth AAAI Symposium on Educational Advances in Artificial Intelligence*, pp. 1210-1217, 2020.