Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a labeled blockchain graph dataset from unspent transaction output and account-based blockchains called Chartalist. The dataset can be used for modeling, analysis, and representation of blockchain that can be used for applications like ransomware payment tracking, price manipulation, and money laundering detection. Chartalist provides transactions and blockchain extraction and graph creation tools for graph-based machine learning methods on the blockchain. The paper proposes 6 ML tasks and does a very good job of motivating and explaining the tasks.
strengths: The paper is well motivated and easy to understand.

The benchmark provides exposure to blockchain and reduces the barrier to entry in the field. The application presented (wash trade detection, address classification, etc.) sounds really interesting.

There is a commitment to maintain and keep it updated. 

The benchmark has dynamic multiplayer graphs with ground truth for anomaly detection.

weaknesses: **Cons/comments/suggestions/questions**

Authors should also include some GNN-based baselines in the work.

Following up on the above IMHO the comparative experiments could be more rigorous.

IIUC the wash trading task is constructed based on a certain heuristic. It may so happen the downstream ML model may learn just the heuristic without learning about the wash trading account distribution. I understand that it is not straightforward to get the ground truth for this. I think the implications of this require some discussion in the paper.

correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: See the weakness section.
review_title: Interest dataset for graph ML in blockchain.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper makes a comprehensive dataset repository for blockchain graph research. It describes the limitations in existing solutions and provides a unified graph creation tool. This paper describes 6 different blockchain graph ML tasks and provides some datasets on Bitcoin, Ethereum, and Dashcoin. This work preprocesses the raw data and prepares the datasets from it, and the ML researchers who do not have enough hardware resources to preprocess the raw data can benefit from this work.

strengths: This work reduces the burden of preprocessing the raw blockchain data which needs a lot of time and resources. And it provides the labeled datasets of different tasks on blockchain graph research and a unified tool to load them.

weaknesses: - For one specific blockchain(like Bitcoin), it doesn't cover all 6 tasks listed in the paper.
- Dashcoin does not have any labeled graph datasets and the task `Coin Tracking Across Blockchains` seems missing.
- The paper only presents the details and baselines of parts of the datasets, and I cannot find the details of other datasets on the website.

correctness: Looks good.

clarity: The paper is well written and provides enough background information for the reader.

relation to prior work: Existing works and their limitations are discussed.

additional feedback: No.
review_title: Chartalist

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposed a set of graph datasets collected from blockchains. The background, importance, and potential applications are discussed. Three tasks are also evaluated with existing methods. The datasets are distinct from the widely used graphs such as citation networks and social networks. However, I wonder if this paper doesn't propose new datasets but gathers some existing graph datasets together. Some datasets are also published in other conferences and available on GitHub. Considering the main contribution of this paper is introducing new datasets but not comparing different existing methods, I think the contribution of this paper is somehow weak.
strengths: 1. The datasets are collected from real-world applications, which include unique patterns.
2. The potential tasks are listed clearly, and the website is well organized.
3. The potential real-world impact is illustrated thoroughly.
weaknesses: 1. Although these datasets are evaluated by some existing methods, the performance of popular supervised/unsupervised architectures is unknown, e.g., GCN and Node2Vec. Is it challenging to evaluate these methods with the proposed graphs? Or do these popular methods fail on the mentioned tasks?
2. ``Some datasets are published somewhere else``. For example, ``Address Classification on Ethereum`` is included in ``AlphaCore: Data Depth based Core Decomposition``; ``Wash-Trading Transaction Identification Dataset`` is published in ``Detecting and quantifying wash trading on decentralized cryptocurrency exchanges``. I think it is okay to include these datasets, but it requires more discussions/experiments other than the published content. 
3. I wonder if graphs' labels are good enough for a fair evaluation of different methods since it is difficult to get the ground truth.
4. I hope more details of datasets could be included in the paper or the supplementary but not only available on the website. 

correctness: The claims are correct since they are included in other published papers.
clarity: More details of the dataset could be included in a table, e.g., how many graphs in total, the size, visualizations of subgraphs, some basic graph properties.
relation to prior work: I think this paper is a summary of some published papers.
additional feedback: I think more details should be included in the paper/supplement but not only available on the website.
The primary concern is that some datasets are published somewhere else. I think this paper should concentrate on what is new but not a summary of existing work like a short review. I appreciate the effort that the authors collect different datasets together. However, I think the contribution is somehow weak to publish a new paper unless there is a significant improvement over the published papers.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes Chartalist which is a platform that hosts a diverse set of datasets from three different cryptocurrency networks (Bitcoin, Ethereum, Dashcoin) that are useful for investigation and conducting research on blockchain networks. The datasets are ready to be used in different ML tasks and facilitate analysis of the transactions on cryptocurrency networks. The datasets contain transaction networks of three different cryptocurrencies and also encompass several different important tasks specific to each network.
strengths: The strength of Chartalist can be summarized as follows:
-	It contains useful datasets from three different cryptocurrency networks that are based on different architectural model (i.e. UTXO vs. account-based).
-	Based on each cryptocurrency network, it suggests several tasks that can be studied on each dataset, providing insights and direction to further research.
-	While there is a considerable lack of labeled data for analyzing blockchain data, Chartalist provides several datasets with labels including different kinds of heist activities.
-	Chartalist is open source, easy to use, and is supposed to be updated regularly.

weaknesses: The paper can be improved on the following aspects:
-	The organization of the paper can be substantially improved. The paper has a major issue with referencing; there is no reference for many of the statements and assumptions. Although many of these statements make sense to an expert in the field, there should be appropriate referencing for consolidating the ideas proposed.
-	The discussion in Section 3.1 does not follow naturally with the paper. Although it is important to discuss the blockchain data type, I think the discussion in this section is very long and does not connect well with the other sections. Also, it does not clearly state why UTXO and account-based architecture are different and why we need to consider the underlying differences when studying samples of these networks. 
-	In Section 3.2, several different tasks have been introduced. However, the explanations for some of these tasks are unclear. It could help to improve the definitions of these tasks and provides some actual examples to better motivate the importance of these tasks.
-	It is mentioned that one important intuition behind Chartalist is to provide datasets for conducting ML/graph-based methods for several important tasks on cryptocurrency networks. However, in the experiments section, most of the baseline methods are basic and simple ML methods, and graph-based methods (such as methods based on GNN) are not considered.
-	Most of the methods are from the previous works of the authors and newer DL methods are not discussed in the evaluation of the datasets and the proposed tasks.
-	The results of the experiments are not discussed comprehensively. For example, it could be nice to discuss why the recall and precision reported in Table 1 are so low for several datasets and methods. Is there any relation between the datasets, method, and performance? Also, the reported results in Table 1 are so low; the paper should discuss whether achieving these low results is useful in any real-world scenario, or whether these kinds of experiments are actually useful in a real scenario?
-	The datasets contain temporal information but no tasks is defined to study the temporal aspect of the data.

correctness: Although there are some assumptions about how the data is crawled from the blockchain (e.g., limiting the time window to be able to tackle the huge data size), the assumptions seem reasonable considering other related research.
clarity: The paper should be improved in at least terms of appropriate referencing.
relation to prior work: The related work should be improved, there are some other related platforms for the blockchain data analysis that have not been considered. One of the important examples is http://xblock.pro/#/ which also has several important datasets that are currently being used by related research. Also, the distinction of Chartalist with other dataset repository could be better explained.
additional feedback: In total, I think Chartalist provides an important source of datasets for facilitating the analysis and investigation of blockchain networks. However, since many future works could consider this paper as an important related work, the paper should be improved in several aspects.
review_title: Chartalist provides a platform that hosts a large set of datasets from different blockchain-based cryptocurrency networks that can facilitate different ML tasks. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors launch Chartalist, a platform to access a variety of datasets from blockchain that can be used for ML research tasks. The datasets are currently compiled from Bitcoin, Ethereum and Dashcoin blockchains. The datasets are well organized and documented in the official website, and easy to use data loaders to pull datasets using Pandas in Python are provided. These datasets can be used for various ML tasks including but not limited to Address Clustering, Address and Transaction Type Classification, Anomalous Transaction Detection etc. Baselines and model comparisons on three such tasks are discussed in the paper. The baselines are helpful to allow researchers get up to the speed with simple modeling approach and allow them to try new ideas to improve the performance. The datasets are partially labeled. The tool and the datasets are open source and authors promise to maintain them. Most of the datasets can be represented in the form of graph network and graph ML techniques can be employed. 

The main contributions and novelty in this work is making a large scale, hard to compile blockchain dataset accessible to the machine learning research community. The datasets can be easily loaded for any analysis or experimentation for ML task using Python. 

strengths: The work published in this paper can have a significant contribution in attracting a wide community of scientists to analyze blockchain data. Nature of pseudo-anonymity of users in cryptocurrencies and a rapid increase in transactional activities on blockchain is attracting cyber criminals to perform ransomware attacks, money laundering or price manipulation. ML has been widely adopted by centralized service providers to prevent cyber crime and so can be used in blockchain based decentralized service providers to make the platforms more secure. These datasets serve the purpose of spurring the required research by ML community on making the blockchain based applications more secure for users. 

The datasets have been made accessible with the help of easy to use data loaders. The authors take accountability in hosting, managing, and updating the data repository at a quarterly basis at-least. The datasets do not contain PII and has no privacy related concern other than what is already available publicly. 
weaknesses: One weakness that is already quoted by the authors in the paper is the potential negative use of the datasets by malicious actors to learn to hide their identities without getting detected. The second weakness of the paper is the lack of details on label generation process beyond the fact that they were sourced from etherscan.io and from past studies. The third weakness is lack of recommended benchmarking metrics for proposed ML tasks. For example, in Table 1, results on standard classification metrics are discussed, but a recommendation around the choice of score threshold to convert soft score to hard labels is missing. 
correctness: To the best of my understanding, the datasets are constructed in a sound way. The transaction network data is curated by connecting blockchain node to P2P network and using ETL tool to parse Bitcoin, Ethereum and Dashcoin. 
clarity: Most of the paper is well written and the authors have provided enough context for the readers so they can understand different blockchain related terminologies wherever required. 

In certain sections, the authors have scope of improvement. For example, Table 3 lacks explanations of results. The number of wash trades that were identified from the first step vs. the second step is not discussed. The analysis depicting results in Table 3 uses proxy labels but how the labels were generated is not clearly explained. As another example, the experimental setup in Section 4.2 is also not well explained. If the number of networks under consideration were reduced to 28, then what defined top 100 networks. 
relation to prior work: It is clearly mentioned in the paper that this is the first attempt to systematically organize blockchain data for the broader ML community. Other methods to analyze blockchain data that include running a Bitcoin client or paying for commercial API are also discussed. Previous contributions are discussed in relations to data repositories, ML benchmark and blockchain analysis tools. 
additional feedback: Suggestions for improvement: 
1) Detailed explanation of label curation in different datasets, 2) Sample datasets in Appendix, 3) Suggested performance metrics for benchmarking.
review_title: Compilation of datasets from blockchains to spur machine learning research for various use cases 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper introduces a collection of blockchain graph datasets and a platform to access them. Several potential applications are explored and assessed. Additional tools are provided to facilitate further data collection.
strengths: The authors are commended for making blockchain data more accessible to those who wish to perform graph ML analysis. This is achieved by providing cleaned and curated blockchain-related datasets, as well as extraction and graph creation tools to facilitate additional data collection. The motivation and methodology for each sample task is well explained.
weaknesses: The paper makes several mentions of Machine Learning benchmarks, including an entire section in the literature review, however the benchmark capability of Chartalist is never outlined or introduced directly in the paper.   
correctness: Yes.
clarity: Yes. The paper is well written and easy to read. 
relation to prior work: Yes.
additional feedback: Excited to see where this project goes.
review_title: A great contribution to the Blockchain ML space

Meta-review: metareview: All reviewers acknowledge the quality of the paper and recommend acceptance, with scores ranging from 6 to 8. The authors have constructively engaged into the discussion with the reviewers, which has contributed to improving the evaluation of some reviewers.

The proposed dataset can have a significant contribution in attracting a wide community of scientists to analyze blockchain data. The datasets is, according to the reviewers, easy to manipulate and has proper documentation.

Ethical considerations were raised by the reviewers and the ethics committee. The authors have answered to these and, in our opinion, have brought the necessary clarifications to remove the concerns.

For these reasons, we recommend acceptance of the paper.
confidence: 2: The area chair is not sure
Decision: Accept