Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: The context of the paragraph at the top of page 8 (starting with "According to Creech and Hu) is not clear. Going back to that paper, it looks like tests were done with both ADFA-LD and KDD-98. Are the numbers quoted in this paragraph for ADFA-LD?

More importantly, it seems like the LSTM proposed in the paper is a little worse than ELM --- the latter doesn't require recurrent neural network training. Why use LSTMs when ELMs work better?

How were the parameters in the LSTM chosen?
review_title: Comparison to other methods?

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose using an LSTM on a sequence of system calls to perform network intrusion detection (NIDS). The idea of using neural networks (in general) for NIDS is old [1]. The idea of using some sort of NN on top of a sequence of system calls for NIDS is published [2]. The idea of using LSTMs for NIDS is published [2]. The paper in [2] operates on counts of N-grams of system calls, rather than on the raw sequence, but that pre-processing does not seem "heavy" to me.

Overall, the proposed system works as well as other proposed NIDS system, and the paper checks portability (which is good).

But, on the con side, I don't see this paper as adding a lot to the state-of-the-art in NIDS. Nor does is the paper well-matched to ICLR. I didn't learn a lot about representations from this paper: many people have thrown LSTM at sequence problems.

Therefore, I think it's below threshold for ICLR. The authors may wish to submit to a security conference.

References:
1. Debar, Herve, Monique Becker, and Didier Siboni. "A neural network component for an intrusion detection system." Research in Security and Privacy, 1992. Proceedings., 1992 IEEE Computer Society Symposium on. IEEE, 1992.
2. Creech, Gideon, and Jiankun Hu. "A semantic approach to host-based intrusion detection systems using contiguousand discontiguous system call patterns." IEEE Transactions on Computers 63.4 (2014): 807-819.
3. Staudemeyer, Ralf C. "Applying long short-term memory recurrent neural networks to intrusion detection." South African Computer Journal 56.1 (2015).
review_title: Incremental advance, not well-suited for ICLR

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents an anomaly-based host intrusion detection method. LSTM RNN is used to model the system-call sequences and the averaged sequence likelihood is then used to determine anomaly, which is the attack. This paper also compares an ensemble method with two baselines as classification model.
+This is is well written and more of ideas are clearly presented.
+It demonstrates an interesting application of LSTM sequential modeling to HIDS problem
-The overall novelty is limited considering the major technical components like LSTM RNN and ensemble method are already established. 
-The contribution of the proposed ensemble method needs further evaluation because it is also possible to use ensemble ideas in kNN and kMC baselines.
review_title: null

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In this paper a novel approach for anomaly detection is considered for the task of intrusion detection based on system call sequence.
The system call sequence is regarded as a language, and multiple lstm-rnn language models are trained and ensembled. 
Diversity in the ensemble is achieved by choosing different hyper parameters for each lstm-LM.
The combination of the LMs is done by averaging transformations of the likelihoods. 

I really like the fact that no attack data is used during training, and I like the LM and ensemble approach. 
The only high level drawbacks I have are the following, which might have a simple answer as I'm not an expert in this field:

- Relaying of system calls seems weak: If the attacker has access to some "normal" sequences of system calls, all she can fool the system by interleaving its malicious system calls with normal ones, in a way that will artificially raise the likelihood of the sequence. 
- A few lines covering other anomaly detection tasks, where RNNs are used, can be added to the introduction, to give a better idea about the novelty of the approach. 

review_title: neat idea, clearly written

Meta-review: This is a pure application paper: an application of LSTMs to host intrusion detection systems based upon observed system calls. And from the application standpoint, I don't believe this is a bad paper, the authors seem to achieve reasonable results from the method (though admittedly, I can't really judge the quality of these results without a lot more familiarity based upon current work in intrusion detection systems). However, in terms of the ICLR audience, I simply don't believe there is enough here to warrant substantial interest. As an example, the authors highlighted the distinction between network intrusion detection systems (NIDS) and host intrusion detection systems (HIDS), and felt that one review was completely unsuitable because they didn't realize this distinction when reading the paper. This is of course a crucial distinction from the security application side, but from the algorithmic/ML side, it's simply not that important, and the fact that there _has_ been previous work exactly on LSTMs for NIDS makes this paper unlikely to have a huge impact in the ICLR community. It would be much better suited to a security conference, where the application could be judged on its own merits, and the community would likely understand much better how significant these results were.
 
 Pros:
 + Nice application of LSTMs to HIDS task
 
 Cons:
 - Nothing really novel from the algorithmic/ML side
 - The significance of the results are difficult to assess without more formal understanding of the problem domains
 
 The PCs have thus decided that this paper isn't ready to appear at the conference.
Decision: Reject