Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: null
review_content: This is a comment from an official ethics reviewer.

The ethical concern raised by the reviewers was on misuse. The authors have structured the dataset appropriately and added sufficient discussion to the paper. There is no reason to hold the paper back for ethical concerns.
review_title: An official ethics review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper provides a new database for graph representation learning. The graphs in the database were generated from function call graphs of malware software. Both the number of graphs and the size of the graphs within the database are significantly larger than existing databases for graph representation learning. Finally, the authors use this new dataset to test existing graph representation learning approaches and propose new potential research directions on this dataset.   

**Update**
There seems to be a consensus for acceptance. I appreciate the author's response to my review. 

strengths: - clearly identify shortcomings in existing datasets
- graph representation is of interest in several application domains so the new dataset can have a significant impact
- hosting via the Georgia Institute of Technology serves provides accessibility
- tests of existing methods as well as proposal of new research directions 
weaknesses: - graphs from different domains can have potentially very different structures, while this is a very large dataset its data comes from a very specific domain
- potential limitations of this limited domain aren't discussed 
correctness: Seems good
clarity: Overall the paper is well written and clearly structured. One small exception, it's not very clearly stated that the dataset is specifically about functions calls from malware. It is in the name of the dataset, but not really in the text. 

relation to prior work: Existing datasets and their limitations are discussed.
additional feedback: The content of Section 2.1. and Table 2 is very redundant. Instead of just restating statistics the space would be used better on describing the differences in the tasks and structures. 

Especially interesting would be a discussion of the potential limitations of this dataset given its specific domain. 

Section 4.1. You mention the time it takes to train the complicated methods, but not the simple ones. For a better comparison, all training times would be interesting.
review_title: Review (updated 19th July 2021)

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper contributes a database for graph representation, Malnet, containing 1.2 million graphs from different classes. The database is comprehensive and has important cybersecurity applications, as the authors describe. The authors use the AndroZoo repository in collecting the Android APK files, extracting then the function call graphs, and using them for different graph representation techniques and model predictions. Two interesting contributions of this dataset are: showing that current techniques in the literature don't capture non-attributed graph topology and showing that diversity inversely correlates with performance (to an extend, intuitive). 
strengths: The paper is strong in the database collected, in its exposition, and in its testing of the dataset. In particular, the online database linked in the paper is quite comprehensive for visualization and testing, and shows interesting applications in cybersecurity.
weaknesses: One concern I have about the study is that this kind of database can also be used for 'bad', e.g. for training malware algs. The authors mention that they numerically relabel the nodes of each graph in order to mitigate potential reverse engineering, but a more in-depth discussion about the potential would strengthen the paper. (The authors also mention this is basically the only such open database).

One part that was confusing to me was in section 2, the explanation for different applications: while the cybersecurity explanation is clear, the other ones just seem to mention existing datasets in other fields, raising the question of how is this dataset adding to those applications, or is that just related work? 
correctness: The database seems soundly constructed, collecting the Android APKs.
clarity: The paper is clearly written, only two minor details:  lines 34-35 are interrupted and line 225 says ‘optmizer'
relation to prior work: The paper does a good job of relating to related datasets.
additional feedback: --
review_title: Graph representation learning dataset for security applications; comprehensive and well-written

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The authors introduce MalNet, a new labelled graph dataset containing over 1 million new function call graphs produced from various kinds of function call graphs found in malicious programs and software. The graphs are labelled into type and family of malware. (Examples of malware types include "Adware", "Trojan" and "Spyware.") The labels are generated from antivirus companies. These call graphs are non-trivial in size, containing an average of thousands of nodes and edges per graph. 
strengths: The paper greatly expands the data available for graph representation learning. The dataset is very rich, containing great variety in graph type and biases that are currently unavailable for research datasets.

The manner in which the dataset was produced is straightforward, and it should be conceptually simple to extend this sort of analysis to other codebases.

There are a number of potential domain-specific applications, including computer security, and program analysis and refactoring.

The exploration of various graph representation techniques suggests several interesting directions for future research, including that the predictive power of very simple graph statistics (like node and edge counts) may be underestimated in the current literature.

The authors provide an easy-to-use website for exploring the dataset, with engaging visual representations of program call graphs. Overall, I am excited about this paper, and would be interested in exploring this data in my own research.
weaknesses: No major noteworthy weaknesses.
correctness: The dataset is constructed in a clear and straightforward manner. The authors train a suite of standard graph learning algorithms on this data to get a sense of performance, including GCNs, LDPs and others. When the authors did not test a standard method (such as NLP-based string embedding methods), they explain clearly the performance bottlenecks preventing them from doing so.
clarity: This paper is exceptionally well-written.
relation to prior work: The authors systematically detail previous datasets for graph representation learning, how the dataset in this paper differs from those, and the kinds of new research their dataset will enable.
additional feedback: Just a couple of small suggestions that would be very nice to have, but are not essential for the publication of this paper:
* A brief comparison of the qualitative properties of these call graphs, and how they might differ from other datasets (e.g. number of connected components, density, etc.)
* A bit of exposition on the properties we might expect various types and family of call graph to have based on prior knowledge of computer security research.
review_title: Great new dataset for graph representations

Decision: Accept