Detecting malicious PDF using CNN
Authors: Raphael Fettaya
Yishay Mansour
Venue:  (2020)
Keywords: Cybersecurity
Convolutional Neural Network
Malware

Abstract: Malicious PDF files represent one of the biggest threats to computer security. To
detect them, significant research has been done using handwritten signatures or
machine learning based on manual feature extraction. Those approaches are both
time-consuming, requires significant prior knowledge and the list of features has
to be updated with each newly discovered vulnerability. In this work, we propose
a novel algorithm that uses a Convolutional Neural Network (CNN) on the byte
level of the file, without any handcrafted features. We show, using a data set
of 130000 files, that our approach maintains a high detection rate (96%) of PDF
malware and even detects new malicious files, still undetected by most antiviruses.
Using automatically generated features from our CNN network, and applying a
clustering algorithm, we also obtain high similarity between the antiviruses’ labels
and the resulting clusters.

Under review as a conference paper at ICLR 2020
DETECTING MALICIOUS PDF USING CNN
Anonymous authors
Paper under double-blind review
ABSTRACT
Malicious PDF files represent one of the biggest threats to computer security. To detect them, significant research has been done using handwritten signatures machine learning based on manual feature extraction. Those approaches are both time-consuming, requires significant prior knowledge and the list of features has to be updated with each newly discovered vulnerability. In this work, we propose novel algorithm that uses a Convolutional Neural Network (CNN) on the byte level of the file, without any handcrafted features. We show, using data set 130000 files, that our approach maintains high detection rate (96%) of PDF malware and even detects new malicious files, still undetected by most antiviruses. Using automatically generated features from our CNN network, and applying a clustering algorithm, also obtain high similarity between the antiviruses' labels and the resulting clusters.
INTRODUCTION
Malware programs still making newspapers' headlines. They used by criminal organizations, governments, and industry to steal money, spy or other unwanted activities. As millions of new malicious samples are discovered every day, spotting them before they harm computer or network remains one of the most important challenges in cybersecurity. During the last two decades, hackers kept finding new attack vectors, giving malware multiple forms. Some use the macros in Office documents while others exploit browser's vulnerabilities with javascript files. This diversity raised a need for new automated solutions.
PDF is one of the most popular types of documents. Despite the lack of awareness of the population, also became an important attack vector for computer systems. Dozens of vulnerabilities are dis- covered every year on Adobe Reader, the most popular software for reading PDF files (I) allowing hackers to take control of the victim's 's computer. PDF malware can be segmented into three main categories: (i) exploits, (ii) phishing and (iii) misuse of PDF capabilities. Exploits operate by taking advantage of a bug in the API of a PDF reader application, which allows the attacker to execute code in the victim's is computer. This is usually done via JavaScript code, embedded in the file. In phishing attacks, the PDF itself does not have any malicious behavior but attempts to convince the user to click malicious link. Such campaigns have been discovered recently (2) and are, by nature, much harder to identify. The last category exploits some regular functionality of PDF files such as running a command or launching Aile.All those attacks can lead to devastating consequences, such as downloading a malicious executable or stealing credentials from a website.
Regardless of recent work in machine learning for malware detection, antivirus companies are still largely focusing on handwritten signatures to detect malicious PDF. This not only requires a lot of human resources but also rarely efficient at detecting unknown variants or zero day attacks. Another popular solution is dynamic analysis by running the files in controlled sandboxed en- vironment. Such approaches increase significantly the chance of detecting new malware, but take much longer and requires access to sandbox virtual machine. They also still require human to define the detection rules according to the file behavior.
In this paper, malicious PDF files. Without any preprocessing of the files, our classifier succeeds to detect 94% of the malicious samples of our test set while keeping False Positive Rate (FPR) at 0.2%. Our classifier outperforms most of the antiviruses (AV) vendors available in the VirusTotal website. We also show that our network can successfully group more than 80% of the malware into different
using Convolutional Neural Network (CNN) in order to detect any type of

---

Under review as a conference paper at ICLR 2020
families agrecing 76% with the name given by the AV. Finally, we will present some examples on which we were able to detect an attack before the AV (zero-day).
To the best of our knowledge, this is the first paper using Convolutional Neural Network to classify malicious PDF files. is also the first one that investigates the ability to automatically classify PDF malware into different families.
Paper organization: We first present the related researches that have been done in machine learn-
ing for detecting malicious PDF and the usage of Deep Learning applied to Malware Detection in
executable files (section 2). We describe how we built our data set in Section 3 and describe our
model in Section 4 We show our results on the data sets in Section 5
We investigate the capability
of our network to differentiate between malware types in Section
Our conclusion is in Section 7
2 RELATED WORK
2.1 MACHINE LEARNING FOR DETECTING MALICIOUS PDF
Signature- based detection used to be the standard cybersecurity, and it is the preferred solution where researchers use signatures identify malicious PDF (I3). However, with the fast increase of threats, the work required by handwritten rules increased significantly, and machine learning has been extensively used in the last decade enrich detection capabilities.
Maiorca, Giacinto and Corona (9) generate features based on keywords (tags) found in the PDF files. They choose the keywords by clustering appearances in benign and malicious sets. They obtain high classification results (99.6% of detection), but their approach mainly focuses on specific types of malware, a subset that does not include exploits and phishing PDF. Munson and Cross 4 define a list of features extracted from both static and dynamic analysis of the PDF files, and train a decision tree classifier, but they used a very small dataset of 87 malicious samples. Tzermias et al. (17) combine static and dynamic analysis, focusing on specific vulnerabilities, and managed to detect 89% of the malware in their dataset. However, it takes 1.5s for their algorithm to run on a single PDF file and requires the usage of VM.
Stavrou and Smutz (15) focus their feature extraction on metadata and structure of the documents. They end up with 202 manually selected features that are used to build Random Forest classifier and obtain a detection of more than 99% and a FPR lower than 0.5%. Sindic and Laskov (16) use an open source PDF parser and retrieve the like structure of the file which they use a feature. They then train some Decision Trees and a SVM on several datasets and manage to detect 95% of new files, keeping an FPR around 0.1%% More recently, Zhang (18) is using Multi-Layer Perceptron (MLP) on 48 manually selected features, and achieves a better detection than 8 antiviruses in the market, obtaining a detection rate of around 95% on their datasets.
We emphasise, that all the previous approaches require prior domain knowledge as they are based on manual feature selection. As far as we know, our work the first to automate the feature extraction of PDF files.
2.2 DEEP LEARNING IN MALWARE DETECTION
In the last years, there have been several efforts in using Deep Learning for detecting malware in executable files (type exe). David and Netanyahu, in DeepSign (5), are using Deep Belief Network with Denoising Autoencoders to automatically generate signatures based on executables' behavior. To do so, they run the files on a sandbox, extract the API calls and create 5000 one-hot encoded features out of them. The signatures consist of 30 features generated by the network that are used in the classifier and achieves a 98.6% of accuracy on their dataset. Pascanu et al. (II) starting from the API calls, are generating an embedding of the malware behavior using Echo State Network and Recurrent Neural Network. They train the network to predict the next API call and use the last hidden state as a feature for classifier. They obtain detection rate of 98.3% with 0. 0.1% of FPR.
Saxe and Berlin (14) are using manually extracted features from static analysis of executables that they use a four-layer perceptron model and detect 95% of the malicious files in their dataset at 0.1 % FPR. Finally, Raff et al., in Malcony, (12) are using Convolutional Neural Network the raw
2

---

Under review as a conference paper at ICLR 2020
binary file. Their results are bit inferior to the others, around 90% accuracy, but no preprocessing is required on the data and predictions are done very efficiently.
DATA SETS
For the experiment, we use 19830 malicious PDF files and 116695 benign ones. The malicious samples are taken from Virus (3). They were uploaded on the website between the 5/5/2018 and the 11/14/2018 They were all detected by at least 10 antiviruses by the 11/20/2018. We also download set of 9300 newer malicious files, following the same rule, to evaluate the degradation with the time. For this set, the files appeared the website between the 2/15/2019 to the 3/15/2019. The benign files were obtained using collaboration with private company. Due privacy issues, we would not be able to make them available. Following the assumptions that the benign PDF evolve much slower than the malicious one, we did not record their date of creation.
We partition our malicious and benign sets into training, validation and test set. In order to make sure our research imitate well as possible the real-word malware classification challenge, the sets were chronologically organized. For the train set, we use 17880 malicious files, from the oldest one the ones that appeared in 9/28/2018, and 105027 benign samples. The validation set helps us tuning the hyperparameters of our model and contains 416 malicious files (until 1//10/2018 18) and 2332 benign. The test set contains the rest.
4 NEURAL NETWORK ARCHITECTURE
We design and present two different models. The first one is minimalist model, with single convolutional layer, followed by global max pool and finishing with a linear layer and a sigmoidal gate, refer to it ModelA. The second model, ModelB, is similar to ModelA, but has addi- tional fully connected layer, right after the global max pool. The additional fully connected layer allows the network to combine the filters together and giving him much more freedom, denote it ModelB. Both the convolutional and fully-connecto layers are directly followed by a ReLU in all the networks.
In order to turn the files' byte code to input vectors, we proceed as follows. Initially, to turn them to fixed size, we select only the first 200kB, if a file is smaller it will be zero-padded at the end. Then, each byte, which is a number between 0 and 255, is mapped to a 16-dimensional vector. To obtain < 255 we simply compute the product W T where x; is a vector
the mapping of the byte i, in 2255 that is equals to at index i and 0 everywhere else, and W E 166x255 is the mapping (or embedding) matrix. This operation is actually the first layer of all our networks and the matrix W is trained by back-propagation together with all the other parameters of our models. Such process commonly referred as learning task-specific embedding, as the matrix W is trained with the network and thus, optimized for our classification problem.
The training has been done for 13 hours on average, on single Nvidia GPU and we ran epochs on the training data.
As regularization technique, we use early-stopping with the help of our validation set. For ModelB we use dropout with p 0.25 after the fully connected layer.
The detailed architecture of those two networks can be found in Appendix B
Due to its simplicity, ModelA cannot employ complex patterns from the data. It only creates filters for some strings, using the convolutional layer, look for them in the file, and combine them for classification. This is very generic and, in many ways, simulates what an antivirus does with signature matching.
ModelB has additional fully- cullecconeetted layer, it is able to combine the output of the filters together. Hence can create multi-string signatures.
3

---

Under review as a conference paper at ICLR 2020
Figure 1
(a) ROC on train set for the models
(b) Same ROC, at different scale
RESULTS
In this section, we will present the results on our dataset. Following the standard malware detection denomination, we refer to malware as positive examples and benign files as negatives. A False Positive (FP) is a benign file that has been detected as malicious and a True Positive (TP) a malicious file that has been correctly detected. Similarly. False Negative (FN) is malicious file that has been detected as benign and a True Negative (TN) a benign file that has been correctly
detected. We will mainly use two metries: the detection rate recall (D) TP+FN and the False FP Positive Rate (FPR) FP4TN- On the first part, we will present the results obtained on the test set, then we will evaluate the degradation of the model on newer files, and we will finish by presenting some examples of (almost) zero-day detection.
TP
Figure 2
(a) ROC on test set for the models
(b) Same ROC, at different scale
5.1
TEST RESULTS
In order to evaluate our algorithm, we compare our results to the antivirus in VirusTotal. As we built our dataset querying for files that have more than 10 detections, small bias is created towards the antivirus but it still helps us understand how well our algorithm work compared to existing solutions. Due to privacy issues, we decided not to upload the benign files to VirusTottll so we cannot evaluate the False Positive Rate of the vendors. For our models, we will take reference for the ranking the detection at a 1% FPR but we will present also the detection at 0.5% and 0.2% FPR. The ROCs on train and test displayed in Figures and The results are presented in Table [
We can see that our best model ranks in the 7th position of all the 52 antiviruses available on the website at the time of the research. At 0.2% FPR models still ranks in the 8th position which demonstrate its reliability.
It is worth mentioning here that most antiviruses use hand-crafted signatures and require significant human resources to accomplish this. In our model, no domain knowledge is required, and the signatures are learned automatically by the model.
4

---

Under review as a conference paper at ICLR 2020
Table 1: Results on models and some AV on the
Rank
Model
Detection
Detection at 0.5%
Detection at 0.2%
McAfee
0.993
6
AVG
0.975
7
ModelBB
0.960
0.953
0,939
7
ModelA 0.961
0.950
0.807
8
GData 0.940
Figure 3: Histogram of the AV detection
30
Number of AV the 5m
Number of the
6
00
0.4
0.6
10
10
Detection ratio
Detection
(a) On the train set
(b) On the test set
We can also note that the ModelB more robust and efficient than the two other, and maintain high detection rate even at a FPR of 0.2%. Finally we remark that ModelA still performs well at I%FPR. This suggests that imitating the AV work, with signature matching, seems like an effective approach for detecting PDF malware.
5.2 DEGRADATION WITH TIME
One of the issues with using machine learning detect malware, is that our detection rate is expected to decrease with time. Indeed, malware are constantly evolving, as hackers keep refreshing their technics, detection datasets become quickly outdated and models performance degrades new samples.
In order to evaluate robustness with time, we download were seen for the first time between the 15th of February and the 15th March 2019. This makes 5 months delay from the oldest file of our training set to this new set. In this experiment, detection for ModelB, using the same threshold of 1% FP, is equal to 97.9% and a bit less for the other (92.7%) The detection (96.0%). This variation can be explained by the fact that our test set is relatively small and maybe more challenging the new In any case, we can observe that our network managed learn patterns that are generic enough not suffer from the time gap.
of 9300 files from VorusTotal that
for the ModelB got even higher than in the original test
interesting to notice that the use of dropout, in the ModelB, increases the chance of generating robust signature. Indeed, running the same network without this operation, does not impact much the detection on the test set (95.8% at 1% FPR), however, the new set, drops to 91.5%. This means that proper regularization forces the network to find more generic identifier for malware, and thus, is less impacted by the time.
5.3
ZERO DAYS
With the same goal of investigating generalization capabilities of our model we also look for files that almost undetected by antiviruses (AV) during our training phase and that started being increasingly detected later. Due to the limitations of the query language in VirusTotal we looked manually for some of those examples.

---

Under review as a conference paper at ICLR 2020
Figure 4: Histogram of the AV detection
(a) Score distribution for the test set and for the new files
(b) Detection of Sonbokli files two weeks after they were discovered
One file we found is labeled as belonging to the Sonbokli family by Microsoft. We downloaded 100 more PDF tagged with this name and having more than 10 antiviruses detecting them in July 2019. Surprisingly, most of these files are presenting the same kind of pattern in Antivirus detection. Initially, a very low number of vendors detect them on the first appearance (usually less than 4, all after January 2019), and, after some times, this number drastically increases, more vendor detect them.
We run our CNN on those 100 files, which detected 75 of them. This means that our CNN was capable learn a signature more robust than the one written by most antiviruses for this specific family of malware. This signature was generic enough to still be relevant 8 months after the end of the training period.
Finally, we look for Sonbokli files of the week of the experiment. We found 42 PDF files uploaded to VirusTotal between the first and the 7th of July 2019, none of them had more than 10 detections, and around half of them detected by Microsoft only. Still, network is able to detect 81% of them as malicious. We show in Figure 4b the number of antiviruses that detect each file, two weeks after them for the first time.
DIFFERENTIATING VARIOUS FAMILIES MALWARE
Our ModelA, which has single convolutional layer followed by global max pool layer, has essen- tially the following semantics. The network passes to the max pool layer list of values indicating the presence of some strings in the file. Those strings, that will call signature by analogy with the signatures generated by the antivirus vendors, are used to perform the classification. The same signatures also (potentially) helpful to distinguish different families of malware.
The main issue while trying to classify malware families is that creating their labels is much more complex. The main reason is that antiviruses' names inconsistent and finding good heuristic for labelling a hard problem (8) For that reason, we decide to use unsupervised learning to investigate how well our CNN can differentiate between different malware families. We will use antivirus labels only as a reference for comparison.
In this experiment, network on them while extracting the output of the global max pool layer. This means that for each sample we will be represented by a vector R 128
retrieve the 1534 malicious samples of our set, then run the ModelA
6.1 VISUALIZATION
We first start by displaying in 2D the vectors obtained after the global max pool layer. To do will use the common dimensionality reduction technique: T-SNE (7). We would like to compare results with antivirus' names. For that, require the antivirus detect the major part of our samples while having enough diversity its labels. The one we found that satisfies the best both of those requirements is Ikarus.

---

Under review as a conference paper at ICLR 2020
For convenience, select only the 15 most common families, which drops only 10% of the sam- ples. We also remove the undetected, which represents around 7% our test. The graph is presented in Figure 3
Figure T-SNE applied to the 2nd layer
Trojan.PDF Phishing
Sggen-Downloader PDF: Pidef
possible-Threst
PDF Explott.POP POF--8
20
Exploit.5S SheliCode
Exploit.1 Pdñka
Trojan.PDF.
Trojan.PDF Agent
20
Explor.PD/--ee NameNe
Exploit CVE-2012-4914
40
Exploit.15
Trojan. Wie32.
ExploitPdfkk
ExplortPDF
-50
We can easily distinguish the Phishing PDF family among the data, from the rest. We can also notice that the Trojan Downloader Pidef family split among few clusters which seem to be different variations of this malware family. Inside few of those groups, appear some files labeled 'Agent', meaning that Ikarus could not assign them to a family although our model seems to be able to do so. Another observation is that the families that are based exploits end up pretty close the graph, those malware programs are exploiting the same kind of vulnerability on the system. By looking more specifically some of these files in VirusTotal, "js-embeddedd" indicating that they run Javascript on the host computer. This indeed strong indicator of maliciousness (6) and explain why those groups next each other.
which is most common
guess here that all
noticed that most of them are labeled
6.2 CLUSTERING
order to dig deeper into the understanding of the network, we run clustering algorithm on the vectors described above, and compare the results with three antiviruses: Ikarus, Microsoft and McAfee. We aim to evaluate two metrics: what part of each cluster has the same label (homogeneity) and how much of each family is contained in specific cluster (completeness).
As a clustering algorithm, we decide to use the density-based algorithm HDBSCAN (IO), as does not require any hyper-parameter except the minimum of files in the cluster, does not make prior assumption on the shape of the clusters and allows some files not to be grouped. We run the algorithm the samples without prior filtering, defining minimum size of 10 files by cluster.
To make the results easily explainable we stick on a simplified definition homogeneity and com- pleteness.
The homogeneity is estimated by the ratio between the most common label in the cluster with the number of element in it. More formally, let L be the set of all the names given by a specific antivirus. The homogeneity H; for cluster C, is given by
H(c, :C
H. max İEL
[a
The completeness is given by the ratio of the biggest family in the cluster that is contain in it. Let i, arg max ((c,8) E C.)l the most common family in C,. The

---

Under review as a conference paper at ICLR 2020
completeness T, of C is given by:
K((l): C.Hl J0 (u)(())):cCC)
T,
Note that, for the computation of the homogeneity do not take into account the files in the clusters that are not detected. The detection rate by cluster and by antivirus can be found in the Appendix [
Figure is displaying our results by clusters and by antivirus for the two introduced metrics. Out of the 1534 files, only 285 did not end up in any of the 14 created clusters. The most common family in each cluster are listed in the Appendix A
Figure 6
(a) Homogeneity by cluster
(b) Completeness by cluster
First, easily notice that, the clusters much more homogeneous than complete. Indeed we ended with few clusters for the Trojan-Downlen
Pidef or for the possible-Threet Acmd. This indicates that algorithm is failing find unique identifier for those families and instead is detecting several variants. We also remark that we have two clusters of exploit (7, 8) with lower level of homogeneity. As suggested in the previous section, our algorithm seems more generic that the AV, and we find few families in each of those clusters without di differentiating them. Finally, an interesting observation the difference between the antiviruses at detecting and differentiating some families. For example, we can notice than in cluster 4, Ikarus labels the files Skeeyah but detects only half of them whether McAfee is using the generic name Artemis but detects of them, this cluster Microsoft does not detect any sample.
7 CONCLUSION
We introduced a Convolutional Neural Network to detects malicious PDF using only the byte level of the PDF files. We have shown that our network achieves a performance comparable to some of the best antivirus in the market without requiring any preprocessing feature extraction. It is able to detect files discovered a few months after its training period and malware that are undetected by most antivirus. Finally, the network proved be capable of distinguishing accurately various malware families.
Using this kind of approach could save significant time of malware analysts and automate the de- tection. It will probably be one of the solutions chosen by antivirus companies to cover the drastic increase of new malware files discovered every day.
In future work, we hope to investigate in additional neural network architectures including Recurrent Neural Network (RNN). Additional future research considering additional meaningful embedding for the raw PDF files, for example, by including some metadata information like tags or xref tables.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
[
https: https://www.com wwtp:w/wwwwwwwww.poototw html 2amm?
[2]
[3] Virus total. http::/wwwwwi.usttt.l..loooo com/
[4] Jesse Cross and Arthur Munson. Deep pdf parsing to extract features for detecting embedded malware. Sandia National Labs, Albuquerque, New Mexico, Unlimited Release SAND2011-7982, 2011.
[5] Omid E David and Nathan S Netanyahu. Deepsign: Deep learning for automatic malware signature generation and classification. In 2015 International Joint Conference Neural Networks (JJCNN, pages 1-8. IEEE, 2015.
[6] Pavel Laskov and Nedim Šmniić. Static detection of malicious srrascript-bearinng pdf docu- ments. In Proceedings of the 27th annual computer security applications conference, pages 373-382. ACM, 2011.
[7] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (NNov) 2008
[8] Federico Maggi, Andrea Bellini, Guido Salvaneschi, and Stefano Zanero. Finding non-trivial malware naming inconsistencies. International Conference on Information Systems Secu- rity, pages 154-159. Springer, 2011.
[9] Davide Maiorca, Giorgio Giacinto, and Igino Corona. A pattern recognition system for mali- cious pdf files detection. In International Workshop on Machine Learning and Data Mining in Pattern Recognition, pages 510-524. Springer, 2012.
[10] Leland Melnnes, John Healy, and Steve Astels. hdbscan: Hierarchical density based clustering.
J. Open Source Software, 2(11))205, 2017.
[11] Razvan Pascanu, Jack W Stokes, Hermineh Sanossian, Mady Marinescu, and Anil Thomas. Malware classification with recurrent networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1916-1920. IEEE, 2015.
[12] Edward Raff, Jon Barker, Jared Sylvester, Robert Brandon, Bryan Catanzaro, and Charles K Nicholas. Malware detection by eating whole exe. In Workshops the Thirty-Second AAAI Conference Artificial Intelligence, 2018.
[13] Sami Rautiainen. A look portable document format vulnerabilities. information security technical report, 34333330 2009.
[14] Joshua Saxe and Konstantin Berlin. Deep neural network based malware detection using two dimensional binary program features. In 2015 10th International Conference Malicious and Unwanted Software (MALWARE), pages 11-20. IEEE, 2015.
[15] Charles Smutz and Angelos Stavrou. Malicious pdf detection using metadata and structural features. In Proceedings of the 28th annual computer security applications conference, pages 239-248. ACM, 2012.
[16] Nedim Šindic and Pavel Laskov. Detection of malicious pdf files based on hierarchical doc- ument structure. In Proceedings of the 20th Annual Network & Distributed System Security Symposium, pages 1-16. Citeseer, 2013.
[17] Zacharias Tzermias, Giorgas Sykiotakis, Michalis Polychronakis, and Evangelos P Markatos. Combining static and dynamic analysis for the detection of malicious documents. Proceed- ings of the Fourth European Workshop System Security, page 4. ACM, 2011.
[18] Jason Zhang. Mipdf: effective machine learning based approach for pdf malware detection. arXiv preprint arXiv: 0888006911, 2018.

---

Under review as a conference paper at ICLR 2020
A CLUSTERING
A.1 DETECTION RATE BY CLUSTERS
Figure Detection rate by clusters by AV
1.0
McMes
fiaauu Microsoft
0.0
rats
0.4
0.2
12
Quster index
A.2
CLUSTER'S S LABEL
Cluster
Ikarus
Microsoft
McAfee
0
possible-Thie Threat. Wwareat.PD........ PDF.Acmd
Pdfdrop
Suspicious-PDF.gen..a
1
possible- Thrcat. Satreat.PDF.... PDF.Acmd
Meterpreter
Artemis
Phish
RDN/Generic.dy
3
Dossible-Threatt.DDF.A.oo
Meterpreter
Artemis
4
Trojan.WWn32..Seeeaah
No detection
Artemis
5
possible-Threail Passible-Threaa.PDD..... Aemd
Meterpreter
Artemis
6
Thojjn-DDownloaderPPDD...PP Pidef
PD..
Domepidief
PDF/Phishing.gen
7
Exploit.PDF-Name
ShellCode
Artemis
8
PDF.Exploi.PPD--IS
Meterpreter
Exploit-PD..bk.ge gen.. Artemis
9
SxpplottWi322..CE2221112222
CVE-2012-4914
10
Trojan-DownllcdderPDPF:...
Domepidief
PDF/Phishing gen.w w
11
Frojjnn-DownlladdeePPPi.... Domepidief
PDI/Phishing gen. w
12
Trooan-DDownlcadee.PPPPPPPc Domepidief
PDF/Phishing.ger
13
Toojan-DDownoodddrPPDF...ded
Domepidief
PDF/Phishing.gen
B NETWORKS ARCHITECTURES
10

---

Under review as a conference paper at ICLR 2020
Figure 8: Model A
1x204800
16x204800
128x1x51197
128x1x1
Embedding
Convolution
Global max pool
Reshape
Linear
Figure 9: Model
1x204800
10x204800
128
Embedding
Convolution
Global max pool
Reshape
Fully connected
Unear
11


Reviews:
[{"review_id": "rJgNHj_RYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a model that detects malicious PDF files. The model applies a Convolutional Neural Network (CNN) to analyze the bytes of the input files. The generated features of CNN achieves good clustering results that are consistent with the ground-truth labels.\n\nThis paper applies an existing CNN architecture. I do not observe any novelty in terms of modeling. I suspect this paper is not interesting to most members of the ICLR community. Therefore, I do not suggest the acceptance of this paper.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lvT8gqYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use CNN to detect malicious PDFs. The paper uses two simple CNN models and train them on a collected dataset, compare with other antivirus software, and conclude that CNN performs better. But I have serious concerns about the experiments.\n\n1. The major concern is with the dataset. The dataset used by this work are collected by authors themselves, and I see a serious problem in this process. The malicious and benign ones are not collected from the same distribution: \"The malicious samples are taken from VirusTotal (3). They were uploaded on the website between the 5/5/2018 and the 11/14/2018\"; \"The benign files were obtained using collaboration with a private company\". Thus, the model may not be actually learning what's malicious and what's benign, but only learning whether the pdf comes from that private company or VirusTotal. It's enough since the test set is also collected this way. It can be fairly easy to distinguish between two different datasets, and the same reasoning applies here. Also, the authors are not making the dataset available due to privacy reasons, which further make the dataset's validity a question. \n\n2. Also, antivirus softwares are applicable on any pdf files, but the model trained with the dataset collected may not be useful in other circumstances. The comparison between them are not fair. It should compare the same model/software but on multiple different datasets to demonstrate the model's general applicability.\n\n3. The experiments done in this work is also not of a satisfactory level. For example, there are some missing blanks in Table 1. Why is that? The figures (e.g., Fig 1/2) can be improved a lot. It did not include baseline and provide little information. The fonts are too small.\n\n4. No major novelty is introduced. The work is an application paper using very simple CNNs on the malicious PDF detection problem. This itself does not make the paper bad but combined with the unconvincing experiments it's a serious weakness.\n\nIn summary, the paper lacks solid experimental results to make its conclusion convincing and its model generalizable. I vote for rejection of the paper.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJglKEDRKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes using CNNs to detect malicious PDFs by analyzing the bytecode. The authors created a dataset of malicious and benign PDFs to be able to train their model and compared against commercial Antivirus.\nThe paper \n\nReject: The paper lacks novelty and have very weak experiments. \n\nThe paper presents a simple application of a small CNN to PDF malicious classification, the main contribution of the paper is the dataset collected, which is not released for privacy reasons.\n\nThe experiments lack strong baselines, and comparisons with previous ML solutions, how important is the design of the CNN for this task.\n\nPrevious work like [12] have already applied the same idea to different but very similar task, malware detection of binary files.\n\nThe paper would be more useful for people working in malware detection if made available as a  tech-report.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission addresses the problem of detecting malicious PDF files. The proposed solution trains existing CNN architectures on a collected dataset and verifies improved performance over available antivirus software. 

There were a number of concerns raised about this work. The main concern the reviewers had with this submission is lack of novelty. The issue is that the paper tackles a standard supervised classification problem which has been extensively explored in the literature and applies an off-the-shelf classification model. Though the particular application has seen less attention in the ICLR community, the problem setting and solution are well known. Thus, the contribution of the work is not sufficient for acceptance. 
Decision: Reject