Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes to automatically recognize domain names as malicious or benign by deep networks (convnets and RNNs) trained to directly classify the character sequence as such.


Pros

The paper addresses an important application of deep networks, comparing the performance of a variety of different types of model architectures.

The tested networks seem to perform reasonably well on the task.


Cons

There is little novelty in the proposed method/models -- the paper is primarily focused on comparing existing models on a new task.

The descriptions of the different architectures compared are overly verbose -- they are all simple standard convnet / RNN architectures.  The code specifying the models is also excessive for the main text -- it should be moved to an appendix or even left for a code release.

The comparisons between various architectures are not very enlightening as they aren’t done in a controlled way -- there are a large number of differences between any pair of models so it’s hard to tell where the performance differences come from. It’s also difficult to compare the learning curves among the different models (Fig 1) as they are in separate plots with differently scaled axes.

The proposed problem is an explicitly adversarial setting and adversarial examples are a well-known issue with deep networks and other models, but this issue is not addressed or analyzed in the paper. (In fact, the intro claims this is an advantage of not using hand-engineered features for malicious domain detection, seemingly ignoring the literature on adversarial examples for deep nets.) For example, in this case an attacker could start with a legitimate domain name and use black box adversarial attacks (or white box attacks, given access to the model weights) to derive a similar domain name that the models proposed here would classify as benign.


While this paper addresses an important problem, in its current form the novelty and analysis are limited and the paper has some presentation issues.
review_title: -

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper applies several NN architectures to classify url’s between benign and malware related URLs.
The baseline is random forests and feature engineering.

This is clearly an application paper. 
No new method is being proposed, only existing methods are applied directly to the task.

I am not familiar with the task at hand so I cannot properly judge the quality/accuracy of the results obtained but it seems ok.
For evaluation data was split randomly in 80% train, 10% test and 10% validation. Given the amount of data 2*10**6 samples, this seems sufficient.
I think the evaluation could be improved by using malware URLs that were obtained during a larger time window.
Specifically, it would be nice if train, test and validation URLs would be operated chronologically. I.e. all train url precede the validation and test urls.
Ideally, the train and test urls would also be different in time. This would enable a better test of the generalization capabilities in what is essentially a continuously changing environment. 

This paper is a very difficult for me to assign a final rating.
There is no obvious technical mistake  and the paper is written reasonably well.
There is however a lack of technical novelty or insight in the models themselves. 
I think that the paper should be submitted to a journal or conference in the application domain where it would be a better fit.

For this reason, I will give the score marginally below the acceptance threshold now.
But if the other reviewers argue that the paper should be accepted I will change my score.


review_title: Technically sound but little innovation/insight w.r.t. models

final_score: rating: 7: Good paper, accept
review_content: 
SUMMARY

This paper addresses the cybersecurity problem of domain generation algorithm (DGA)  detection. A class of malware uses algorithms to automatically generate artificial domain names for various purposes, e.g. to generate large numbers of rendezvous points. DGA detection concerns the (automatic) distinction of actual and artificially generated domain names. In this paper, a basic problem formulation and general solution approach is investigated, namely that of treating the detection as a text classification task and to let domain names arrive to the classifier as strings of characters. A set of five deep learning architectures (both CNNs and RNNs) are compared empirical on the text classification task. A domain name data set with two million instances is used for the experiments. The main conclusion is that the different architectures are almost equally accurate and that this prompts a preference of simpler architectures over more complex architectures, since training time and the likelihood for overfitting can potentially be reduced.

COMMENTS

The introduction is well-written, clear, and concise. It describes the studied real-world problem and clarifies the relevance and challenge involved in solving the problem. The introduction provides a clear overview of deep learning architectures that have already been proposed for solving the problem as well as some architectures that could potentially be used. One suggestion for the introduction is that the authors take some of the description of the domain problem and put it into a separate background section to reduce the text the reader has to consume before arriving at the research problem and proposed solution.

The methods section (Section 2) provides a clear description of each of the five architectures along with brief code listings and details about whether any changes or parameter choices were made for the experiment. In the beginning of the section, it is not clarified why, if a 75 character string is encoded as a 128 byte ASCII sequence, the content has to be stored in a 75 x 128 matrix instead of a vector of size 128. This is clarified later but should perhaps be discussed earlier to allow readers from outside the subarea to grasp the approach.

Section 3 describes the experiment settings, the results, and discusses the learned representations and the possible implications of using either the deep architectures or the “baseline” Random Forest classifier. Perhaps, the authors could elaborate a little bit more on why Random Forests were trained on a completely different set of features than the deep architectures? The data is stated to be randomly divided into training (80%), validation (10%), and testing (10%). How many times is this procedure repeated? (That is, how many experimental runs were averaged or was the experiment run once?).

In summary, this is an interesting and well-written paper on a timely topic. The main conclusion is intuitive. Perhaps the conclusion is even regarded as obvious by some but, in my opinion, the result is important since it was obtained from new, rather extensive experiments on a large data set and through the comparison of several existing (earlier proposed) architectures. Since the main conclusion is that simple models should be prioritised over complex ones (due to that their accuracy is very similar), it would have been interesting to get some brief comments on a simplicity comparison of the candidates at the conclusion.

MINOR COMMENTS

Abstract: “Little studies” -> “Few studies”

Table 1: “approach” -> “approaches”

Figure 1: Use the same y-axis scale for all subplots (if possible) to simplify comparison. Also, try to move Figure 1 so that it appears closer to its inline reference in the text.

Section 3: “based their on popularity” -> “based on their popularity”


review_title: A well-written paper that compares five deep architectures for the cybersecurity task of domain generation algorithm detection

Meta-review: meta score: 4

This is basically an application in which some different deep learning approaches are compared on the task of automatically identifying domain names automatically generated by malware.  The experiments are well-constructed and reported.  However, the work does not have novelty beyond the application domain, and thus is not really suitable for ICLR.

Pros
 - good set of experiments carried out on an important task
 - clearly written
Cons
 - lacks technical novelty

Decision: Reject