Title: LSTM-Based System-Call Language Modeling and Ensemble Method for Host-Based Intrusion Detection
Authors: Gyuwan Kim
Hayoon Yi
Jangho Lee
Yunheung Paek
Sungroh Yoon
Venue: ICLR 2017

Abstract: In computer security, designing a robust intrusion detection system is one of the most fundamental and important problems. In this paper, we propose a system-call language-modeling approach for designing anomaly-based host intrusion detection systems. To remedy the issue of high false-alarm rates commonly arising in conventional methods, we employ a novel ensemble method that blends multiple thresholding classifiers into a single one, making it possible to accumulate `highly normal' sequences. The proposed system-call language model has various advantages leveraged by the fact that it can learn the semantic meaning and interactions of each system call that existing methods cannot effectively consider. Through diverse experiments on public benchmark datasets, we demonstrate the validity and effectiveness of the proposed method. Moreover, we show that our model possesses high portability, which is one of the key aspects of realizing successful intrusion detection systems.


Review discussion:
The context of the paragraph at the top of page 8 (starting with "According to Creech and Hu) is not clear. Going back to that paper, it looks like tests were done with both ADFA-LD and KDD-98. Are the numbers quoted in this paragraph for ADFA-LD?
More importantly, it seems like the LSTM proposed in the paper is a little worse than ELM --- the latter doesn't require recurrent neural network training. Why use LSTMs when ELMs work better?
How were the parameters in the LSTM chosen?
Thank you for your evaluation efforts and questions.
Q1) Are the numbers quoted in this paragraph for ADFA-LD?
A1) Yes, these numbers are for the ADFA-LD dataset, as mentioned in Section 3.2 of our paper.
Q2) Why use LSTMs when ELMs work a little better?
A2) In our opinion, our LSTM-based approach should be much easier to develop and to adapt to new attacks.  As many other recent applications of deep learning, no feature extraction was needed in our work to match the performance of the ELM approach that requires heavy feature extraction efforts.
It is true that RNN training takes (mostly computing) time, but it would often take more time (and often creativity by experts) to extract a good set of features that lead to satisfactory downstream performance.
Consequently, for ELM-style approaches, it would take nontrivial efforts to develop a new set of additional features that can cope with new types of attacks.
Furthermore, various assumptions need to be made in order to use ELM features unlike our LSTM-based end-to-end framework that does not need such assumptions. Taken together, our approach can be adapted to new attacks more easily since it requires no feature extraction and strong assumptions.
Another important point is that our methodology is not orthogonal to existing feature-based IDS algorithms but can evolve together by reflecting the features known to the community as absolutely useful features, although the current paper emphasized more on deep learning aspects in order to clearly deliver the advantages of LSTM-based methodology to the readers.
Q3) How to choose LSTM parameters?
A3) We set aside test data first and used the remaining data for training and cross validation to determine the best (most importantly in terms of perplexity) set of parameters on the grid of parameters. We used the test data to report the final performance.
In this paper a novel approach for anomaly detection is considered for the task of intrusion detection based on system call sequence.
The system call sequence is regarded as a language, and multiple lstm-rnn language models are trained and ensembled.
Diversity in the ensemble is achieved by choosing different hyper parameters for each lstm-LM.
The combination of the LMs is done by averaging transformations of the likelihoods.
I really like the fact that no attack data is used during training, and I like the LM and ensemble approach.
The only high level drawbacks I have are the following, which might have a simple answer as I'm not an expert in this field:
- Relaying of system calls seems weak: If the attacker has access to some "normal" sequences of system calls, all she can fool the system by interleaving its malicious system calls with normal ones, in a way that will artificially raise the likelihood of the sequence.
- A few lines covering other anomaly detection tasks, where RNNs are used, can be added to the introduction, to give a better idea about the novelty of the approach.
Thank you for your constructive comments.
The type of attack mentioned in your first comment ("Relaying of system calls ...") is related to the mimicry attack (Wagner and Soto, 2002), which may allow an attacker to cloak the intrusion to avoid detection by intrusion detection systems. Completely avoiding a general form of mimicry attack remains an open problem, but as we briefly mentioned in the Conclusion section, we are planning to consider the possibility of applying our LSTM-based language modeling approach to such attacks. Using our LSTM-based method allows us to estimate the likelihood of long system-call sequences, which may be helpful for analyzing the capability of handling mimicry attacks to some extent.
However, there is currently no public benchmark or dataset for testing mimicry attacks, and in-depth analysis of an algorithm to detect mimicry attacks will be very challenging. In the revision, we will include relevant backgrounds and discussions in the manuscript.
As you pointed out, there exist RNN-based anomaly detection methods in the literature (e.g., Cannady, 1998; Ryan et al., 1998; Mukkamala et al., 2002; Wang et al., 2010). Compared with these approaches, the most salient advantage of our approach is the use of a deep learning-based language modeling of system call sequences in combination with ensemble methods for end-to-end learning and robust intrusion detection. We will include additional contents on related work in the revised manuscript in order to put our approach in a proper context and contrast our method with existing work.
The authors propose using an LSTM on a sequence of system calls to perform network intrusion detection (NIDS). The idea of using neural networks (in general) for NIDS is old [1]. The idea of using some sort of NN on top of a sequence of system calls for NIDS is published [2]. The idea of using LSTMs for NIDS is published [2]. The paper in [2] operates on counts of N-grams of system calls, rather than on the raw sequence, but that pre-processing does not seem "heavy" to me.
Overall, the proposed system works as well as other proposed NIDS system, and the paper checks portability (which is good).
But, on the con side, I don't see this paper as adding a lot to the state-of-the-art in NIDS. Nor does is the paper well-matched to ICLR. I didn't learn a lot about representations from this paper: many people have thrown LSTM at sequence problems.
Therefore, I think it's below threshold for ICLR. The authors may wish to submit to a security conference.
References:
1. Debar, Herve, Monique Becker, and Didier Siboni. "A neural network component for an intrusion detection system." Research in Security and Privacy, 1992. Proceedings., 1992 IEEE Computer Society Symposium on. IEEE, 1992.
2. Creech, Gideon, and Jiankun Hu. "A semantic approach to host-based intrusion detection systems using contiguousand discontiguous system call patterns." IEEE Transactions on Computers 63.4 (2014): 807-819.
3. Staudemeyer, Ralf C. "Applying long short-term memory recurrent neural networks to intrusion detection." South African Computer Journal 56.1 (2015).
Thank you for your review efforts.
Also, thank you for appreciating the advantage of our approach in terms of portability, which is a key issue in the problem this paper is trying to address.
Nonetheless, regarding your “con side” comments, I have to point out that this paper is NOT on NIDS. This paper is host-based intrusion detection system (HIDS), which is different from NIDS in multiple aspects. I am afraid that the reviewer may have started with this misunderstanding of the topic of this paper, leading to further confusions about our contributions.
Recently, many successful applications of LSTMs and other memory-equipped RNN units are being reported in various domains. In many cases, LSTM-based algorithms are not trivial to develop, especially when one targets at beating the previous state-of-the-art results.
Moreover, using deep networks allows us to discover new insights into the problem that would otherwise be difficult to uncover using conventional “shallow” methods. Often, such additional insights are related to the representation learning capability of deep networks.
Therefore, it is rather difficult to agree with your opinion that using LSTMs to solve a novel application does not deserve an ICLR paper. Our LSTM-based solution to HIDS also required a non-trivial process for its development and provides the above advantages of deep learning, including its representation learning capability (e.g., see Figure 5, which shows the representation of system calls).
Furthermore, our method employs a ensemble method for robustly complementing the LSTM component. We did not simply apply LSTM units to HIDS and claim a contribution. We thoroughly analyzed the problem, formulated an LSTM-based solution in a language-modeling framework, blended it with an ensemble method for robustness, and validated our method with well-known benchmarks. I believe that machine learning researchers will be interested in knowing how using an LSTM-based approach can solve a new application in security, even without tedious pre-processing and feature engineering.
Regarding your citation of the two previous papers on NIDS, I appreciate your mentioning them, but, again, some of them are more on NIDS.
Furthermore, your mentioning “The idea of using LSTMs for NIDS is published [2].” is wrong, I suppose. I perused this paper but could not find any mentioning about LSTM at all.
Still, I appreciate your mentioning these papers and will try to revise the manuscript with comparisons with them.
Additionally, you mentioned that “pre-processing does not seem “heavy” to me,” but this is also very difficult to agree with. According to our experience in industrial projects with  data centers, the burden of pre-processing is typically very high. Terabytes of log streams are being produced every day, and it typically takes a vast amount of time and human efforts to pre-process them for downstream analyses based on traditional methods. One of our key contributions is to apply deep-network approach to relieve this burden of pre-processing of logging data. In the revision, I will also try to mention the difficulty of pre-processing in terms of HIDS, emphasizing the benefits of removing it.
Please post additional questions and comments if some arises. I will be more than happy to answer them. Thank you again for your reviewing efforts.
Sorry, I meant HIDS not NIDS. I also had a typo .. I meant to say "The idea of using LSTMs for NIDS is published [3]." All other uses of NIDS should read "HIDS".
This paper presents an anomaly-based host intrusion detection method. LSTM RNN is used to model the system-call sequences and the averaged sequence likelihood is then used to determine anomaly, which is the attack. This paper also compares an ensemble method with two baselines as classification model.
+This is is well written and more of ideas are clearly presented.
+It demonstrates an interesting application of LSTM sequential modeling to HIDS problem
-The overall novelty is limited considering the major technical components like LSTM RNN and ensemble method are already established.
-The contribution of the proposed ensemble method needs further evaluation because it is also possible to use ensemble ideas in kNN and kMC baselines.
Thank you for your comments.
C1) Novelty
R1) The authors believe that the key contribution of this paper should be found in the proposal of using LSTM-based language modeling of system calls for end-to-end learning of anomalies and a robust ensemble method for reducing false alarm rates. These components allowed us to design a very effective methodology for addressing the important problem of HIDS. As is clearly shown in the paper, we did not simply combine existing components, blindly applying them to a new problem. We carried out thorough analyses of the problem, devised an effective and novel solution method, and performed various experimental studies to demonstrate the effectiveness of our approach.
C2) Ensemble methods
R2) We already mentioned why it would be inappropriate to use ensemble ideas in the kNN and kMC in Section 3.2 (the last paragraph on page 7). Compared with existing ensemble methods, our approach could significantly lower the false alarm rate, which is one of the most important performance metrics in the IDS context. We expect that we can apply our approach to other threshold-based classifiers.
In order to reflect the comments from reviewers, the revised manuscript now includes additional references to related work (Introduction) and more in-depth comparison with existing methods (Section 3.2) as well as more details of our future work (Conclusion).

Meta-review:
role: PC
data: comment: This is a pure application paper: an application of LSTMs to host intrusion detection systems based upon observed system calls. And from the application standpoint, I don't believe this is a bad paper, the authors seem to achieve reasonable results from the method (though admittedly, I can't really judge the quality of these results without a lot more familiarity based upon current work in intrusion detection systems). However, in terms of the ICLR audience, I simply don't believe there is enough here to warrant substantial interest. As an example, the authors highlighted the distinction between network intrusion detection systems (NIDS) and host intrusion detection systems (HIDS), and felt that one review was completely unsuitable because they didn't realize this distinction when reading the paper. This is of course a crucial distinction from the security application side, but from the algorithmic/ML side, it's simply not that important, and the fact that there _has_ been previous work exactly on LSTMs for NIDS makes this paper unlikely to have a huge impact in the ICLR community. It would be much better suited to a security conference, where the application could be judged on its own merits, and the community would likely understand much better how significant these results were.
 
 Pros:
 + Nice application of LSTMs to HIDS task
 
 Cons:
 - Nothing really novel from the algorithmic/ML side
 - The significance of the results are difficult to assess without more formal understanding of the problem domains
 
 The PCs have thus decided that this paper isn't ready to appear at the conference.

Decision: Reject