BlockFound: Customized blockchain foundation model for anomaly detection

Authors: Jiahao Yu, Xian Wu, Hao Liu, Wenbo Guo, Xinyu Xing
Primary area: foundation or frontier models, including LLMs
Keywords: Large language model, blockchain, anomaly detection

Abstract:
We propose BlockFound, a customized foundation model for anomaly blockchain transaction detection. 
Unlike existing methods that rely on rule-based systems or directly apply off-the-shelf large language models, BlockFound introduces a series of customized designs to model the unique data structure of blockchain transactions. 
First, a blockchain transaction is multi-modal, containing blockchain-specific tokens, texts, and numbers. 
We design a modularized tokenizer to handle these multi-modal inputs, balancing the information across different modalities. 
Second, we design a customized mask language learning mechanism for pretraining with RoPE embedding and FlashAttention for handling longer sequences.
After training the foundation model, we further design a novel detection method for anomaly detection. 
Extensive evaluations on Ethereum and Solana transactions demonstrate BlockFound's exceptional capability in anomaly detection while maintaining a low false positive rate. 
Remarkably, BlockFound is the only method that successfully detects anomalous transactions on Solana with high accuracy, whereas all other approaches achieved very low or zero detection recall scores.
This work not only provides new foundation models for blockchain but also sets a new benchmark for applying LLMs in blockchain data.

Decision: reject
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729608536036

thread_type:
official_review

<thread_object>
creator: reviewer_nnuh

<content>
summary:
This paper introduces BlockFound, a customized foundation model designed for detecting anomalous blockchain transactions. It modifies BERT model by using RoPE embedding and FlashAttention to suit the characteristics of blockchain transactions. The performance of BlockFound is evaluated using real Ethereum and Solana transaction data, demonstrating its superiority on key performance metrics over baseline methods.

strengths:
First, the idea of using a LLM to detect anomalous transactions in blockchain is innovative. 
Second, the authors have appropriately adapted the LLM for blockchain transaction data, and these modifications have been proven effective through ablation study. 
Third, the authors evaluated BlockFound using actual blockchain data, and the results demonstrate that BlockFound achieves better performance compared to baseline methods.

weaknesses:
1. The introduction of this paper needs to sufficiently explain the motivation behind utilizing an LLM for detecting anomalous transactions in blockchain environments. It lacks evidence to support the claim that blockchain attacks are mainly due to anomalous transactions. Indeed, existing works have discussed that vulnerabilities in consensus protocols can lead to attacks authors mentioned, including double-spending attacks, even when transactions are valid and normal. From another perspective, even if anomalous transactions are detected, the paper does not provide evidence demonstrating that such detection can effectively prevent attacks. Additional references or evidence should be provided to justify the need for anomaly detection in blockchain transactions.
2. The paper needs to clearly define what is an anomalous transaction. The ground truth for such anomalies is not clear. Only with a known ground truth of anomalous transactions can we discuss whether "using the reconstruction errors as the metric for anomaly detection" is appropriate.
3. There is a need for more detailed analysis on the reliability of the datasets used (types, the number of training samples, the number of testing samples, etc.). For example, the statement "our Ethereum dataset consists of 3,383 benign transactions for training, 709 benign transactions for testing, and 10 malicious transactions. The data was collected from October 2020 to April 2023." raises questions. Does this imply there were only ten malicious transactions on Ethereum from October 2020 to April 2023, or were these ten selected by the authors? If selected, what were the criteria for their selection?
4. Why did the study use two different model architectures for different datasets? Does this indicate that the learning models for malicious transaction patterns lack transferability, necessitating the independent training of a model for each specific blockchain/dataset?
5. While the paper is clear and logically presented, it is more like a technical report than an academic paper.

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536799498

thread_type:
official_review

<thread_object>
creator: reviewer_im5b

<content>
summary:
This paper presents BlockFound, a foundational model customized for detecting anomalies in blockchain transactions. It utilizes a modular tokenizer to handle multimodal inputs and employs masked language learning, RoPE embeddings, and FlashAttention technology for processing long sequence data. Extensive evaluations of transaction data from Ethereum and Solana demonstrate BlockFound's exceptional capabilities in anomaly detection while maintaining a low false positive rate.

strengths:
1. BlockFound is a foundational model tailored for detecting anomalies in blockchain transactions, designed to accommodate their unique data structures.
2. A modular tokenizer processes multimodal inputs—specific tokens, text, and numbers—enhancing the accuracy of transaction feature capture.
3. Utilizing masked language learning, RoPE embeddings, and FlashAttention, the model effectively handles long sequence data, improving its ability to process lengthy transaction records.
4. Experimental evaluations on Ethereum and Solana transactions show that BlockFound excels in anomaly detection, demonstrating high accuracy and low false positive rates.
5. Design for the DeFi environment, offering valuable detection methods to protect user assets and enhance the security of blockchain financial transactions.

weaknesses:
1. The paper primarily conducts experimental evaluations on Ethereum and Solana networks, lacking validation of the model's generalization capability and adaptability.
2. The paper highlights the model's performance but lacks transparency in decision-making. Integrating tools could improve interpretability and enhance user trust.
3. The BlockFound model may need further adjustments to meet privacy standards.

questions:
1. How does fine-tuning the GPT-4 model using the API enhance its performance for the same tasks or domains?
2. Can the model's generalization capability be further validated on other blockchain platforms?
3. As the blockchain ecosystem evolves, new attack patterns and transaction types may emerge. Can the model adapt to these changes?
4. Does noise and inaccurate information in the data significantly impact the model's performance?
5. In handling blockchain transaction data, privacy protection is a crucial consideration. How does the article address and safeguard user privacy data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730630238681

thread_type:
official_review

<thread_object>
creator: reviewer_h1p4

<content>
summary:
This paper proposes a deep-learning model for anomaly blockchain transaction detection. By designing a modularized tokenizer for blockchain-specific tokens, texts, and numbers in the transactions, the model handles these multi-modal inputs, balancing the information across different modalities. The model also adapts the RoBERT to train the foundation model with a mask language learning, and makes use of the encoder to detect anomaly transactions.
The evaluation of the model on a meticulously curated dataset demonstrates its potential for applications, showcasing practical applicability in specific environments.

strengths:
1.The paper designs a novel tokenizer that deal with different types of multi-modal inputs in a transaction record, which capture the sematic in different fields effectively and shows some performance improvement;
2.The paper adopts BERT-like model and use
mask language modeling to train the foundation model rather than using GPT-style models, forming a lighter model architecture.
3.The paper evaluates the proposed model in real-world transaction data, and the results appear to be quite promising, especially on Solana with high accuracy.

weaknesses:
1.Although the adapted foundation model shows some performance improvement with RoPE and FlashAttention, it represents a typical data mining method that has been extensively applied to various tasks, thus limiting its originality.

2.The authors rank all the addresses in transactions by frequency and retain the top 7,000 most frequent addresses. Obviously, some information is lost, and there is no guarantee that low-frequency addresses and transactions are necessarily benign.

3.FlashAttention is a key module to handle long inputs in the foundation mode. However, there has been no experimental evidence that this approach has any impact on the accuracy or complexity of the model.

4.There are several Embeddings as inRoBERT such as Position Embedding + Token Embedding and Segment Embedding. However, the authors does not describe how many embeddings are adopted and what is the difference between these embeddings used in their model and those we generally known in RoBERT.

5.The authors should clarify why specific methods, such as RoBERT, were chosen for the framework since there are a number of other BERT-like models that can be used for the modelling of long sequences, such as ELECTRA.

6.Some seem too subjective as no relevant cases or references were found, for example: “..these models cannot capture
the long-range dependencies and complex temporal dynamics inherent in transaction data, resulting sub-optimal modeling performance...” in Section I and "...using MLM can significantly reduce
the computational cost...." in Section III.B

questions:
1.What is the time and space complexity of the proposed method, and how does FlashAttention affect it?

2.Are low-frequency addresses BENIGN? How well does the model work if low frequency addresses are retained?

3.Why does the model chose RoBERT as foundation model and not another model? 

4.In Tokenizer, all tokens share the same vocabulary, since unique hash addresses are individual tokens, how can the size of this vocabulary be controlled to make it available at all times?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696869373

thread_type:
official_review

<thread_object>
creator: reviewer_3gbo

<content>
summary:
The paper proposes BlockFound, a customized foundation model for anomaly transaction detections in Blockchain. Traditional rule-based approaches and techniques relying on off-the-shelf language models struggle with generalizability and scalability. In order to address previous limitations, BlockFound leverages a multi-modal tokenizer to handle the transaction features along with a Bert-based architecture using mask language modeling (MLM) to reduce computational demands. The experiment results demonstrate BlockFound outperforms the baseline models by achieving higher accuracy with lower false positive rates.

strengths:
- Clear justification for using multi-modal approach to process input Blockchain transactions.

- Comprehensive evaluation between BlockFound and baseline models with ablation studies for validating the effectiveness of the designs.

- The implementation of the model and data are open-sourced.

weaknesses:
- The definition of anomaly transactions is ambagious. 

- The size of malicious transaction data is limited.

- Potential redundancy in transaction data. Since contract templates (e.g., ERC-20 tokens) are wildly used in smart contract development, the dataset may contain duplicate transactions.

questions:
I appreciate authors’ efforts in identifying anomaly transactions in Blockchain via machine learning-based techniques. The methodology is clear and the model design is persuasive in general, but I still have several concerns on the transaction data used in this manuscript along with questions for authors to respond.

## Q1: The definition of anomaly transactions is ambagious.
Since the blockchain transactions can vary wildly in structure and content, it would be helpful to have a clear definition of the anomaly transactions. From my understanding, there could be various kinds of transactions regarded as anomaly. For example, the transactions emitted by malicious users to trigger smart contract vulnerabilities either on the implementation level (e.g., integer overflow and reentrancy) or logic level (e.g., inappropriate access control to methods) can be one kind of anomaly transactions. The fraud/phishing transactions belong to another kind. Could the authors supplement a clear definition of the anomaly transactions studied in this paper?

## Q2: The size of malicious transaction data is limited.

I noticed that only 28 malicious transactions in total were used in the training/testing dataset, which is quite limited. I am concerned about if the model can effectively learn the features of anomaly transactions from such a small dataset. Could the authors justify how the proposed model can reliably learn and differentiate the benign and malicious transactions based on the imbalanced dataset?

## Q3: Potential redundancy in transaction data.

Contract templates are wildly used in real-world smart contract development, leading to different smart contracts may offering similar or even identical APIs to the users. This could cause potential duplication in the transaction data. I have two related questions for the authors:

- Q3.1: For the transactions calling to the same methods with different argument values, do they contribute the equally to model training, i.e., can they be considered as duplicate?

- Q3.2: If the above transactions are indeed duplicate, is it possible that similar redundant transactions are present in the dataset used for this study?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731995153034

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
**Rebuttal Summary**

We thank the reviewers for their constructive feedback. Below, we summarize the new experiments and changes made in the revised paper:

----------

### **New Experiments**
1.  **Impact of FlashAttention (Reviewer h1P4)**
    
    -   Conducted an ablation study to evaluate training time and memory efficiency with and without FlashAttention.
    -   Results: FlashAttention reduced training time (e.g., Ethereum: 9,415s → 7,024s) and nearly halved GPU memory usage, enabling training for Solana.
2.  **Choice of RoBERTa (Reviewer h1P4)**
    
    -   Conducted experiments with DeBERTa to validate that the choice of base model has minimal impact on performance, emphasizing framework agnosticism.
3.  **Fine-tuning GPT-4o (Reviewer im5B)**
    
    -   Performed fine-tuning on GPT-4o using Ethereum data to assess anomaly detection performance. Results showed limited improvement over direct GPT-4 use, highlighting constraints of the current fine-tuning API.
4.  **Evaluation of noisy data (Reviewer im5B)**
    
    -   Simulated the inclusion of malicious samples in the benign training set and conducted an ablation study to assess robustness under noise rates of 50%
    -   Added results to show that the model maintains reasonable performance despite noise, demonstrating resilience.
----------
Below, we also summarize the key points in our responses:

### Key Points in Our Responses

**Reviewer 3gbo**

1.  We clarified the definition of anomalous transactions, emphasizing their relevance to deviations from typical patterns and their potential to cause financial losses.
2.  We clarified the concerns about the dataset size, explaining the careful selection of verified malicious transactions from reputable sources like CertiK and PeckShield.
3.  We analyzed potential redundancy in the transaction data and demonstrated that duplication is minimal, with only 0.05% of transactions showing high BLEU similarity.

----------

**Reviewer h1P4**

1.  We justified the use of RoPE and FlashAttention, explaining their necessity for handling long transaction sequences, and highlighted our main technical contribution is the introduction of multi-modal tokenization and a post-training detection approach.
2.  We clarified tokenization strategies to highlight the trade-offs between computational efficiency and information retention.
3.  We conducted additional experiments to evaluate the impact of FlashAttention on time and memory efficiency, showing significant improvements.
4.  We clarified the embedding structures used in our approach.
5.  We demonstrated that the choice of base model, like DeBERTa or RoBERTa, does not significantly affect framework performance, supporting model-agnostic applicability.

----------

**Reviewer im5B**

1.  We discussed the generalization concerns, explaining the selection of Ethereum and Solana networks and our future work.
2.  We emphasized our commitment to data privacy, using only public blockchain data and hashed identifiers to safeguard sensitive information.
3.  We discussed interpretability as a key future direction to enhance user trust and transparency, leaving it as future work.
4.  We demonstrated the model’s adaptability to evolving blockchain ecosystems by suggesting periodic retraining and validating its performance on post-training data.

----------

**Reviewer NnUh**

1.  We added references in the introduction to elaborate on the motivation for anomaly detection in blockchain transactions.
2.  We clarified the definition of anomalous transactions and their ground truth, explaining the verification process for malicious transactions used in the dataset.
3. We clarified the malicious transaction selection process.
4.  We clarified the differences in model architectures for Ethereum and Solana datasets, explaining the rationale behind these choices based on dataset size and overfitting risks.
5.  We highlighted our contribution as a research paper in studying anomaly detection for blockchain transactions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733260511468

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal and Discussion Summary

comment:
Dear Reviewers and AC,

We sincerely thank all the reviewers for their insightful comments and recognition of our work. We are encouraged that the reviewers have highlighted the following strengths in our paper:

-   **Innovative approach to blockchain anomaly detection using large language models** (Reviewers im5B, NnUh)
-   **Effective use of a multi-modal tokenizer** to process blockchain transactions, capturing semantics across different modalities (Reviewers 3gbo, h1P4, im5B)
-   **Comprehensive evaluation** with real-world data and ablation studies validating the effectiveness of our designs (Reviewers 3gbo, h1P4, NnUh)
-   **Successful adaptation of transformer architectures**, including RoPE embeddings and FlashAttention, to handle long transaction sequences (Reviewers h1P4, im5B)
-   **Promising results in anomaly detection** with high accuracy and low false positive rates, especially on the Solana dataset (Reviewers h1P4, im5B, NnUh)

Below, we list the main concerns raised by the reviewers and highlight how we have addressed them during the discussion period.

----------

**Reviewer 3gbo:**

-  We clarified the definition of anomalous transactions in our setting and emphasized that our model is trained solely on benign data, with malicious transactions used only for testing. 
----------

**Reviewer h1P4:**

-   We highlighted that our main technical contributions include multi-modal tokenization and post-training detection. While the reviewer did not explicitly respond, we believe our explanation adequately addressed their concerns.
    
-   We conducted an ablation study showing significant improvements in training time and memory efficiency with FlashAttention, which was included in the revised paper. 
    
-    We explained the trade-offs between computational efficiency and information retention in retaining only the top 7,000 addresses. 
    
-   We provided experimental evidence demonstrating that the framework's performance is agnostic to the choice of base model. 
    
-   We revised the paper to include relevant references and corrected subjective claims, as suggested. 
    
----------

**Reviewer im5B:**
    
-   We detailed experiments with fine-tuning GPT-4, highlighting limitations due to API constraints and costs. The reviewer acknowledged this response but maintained their original rating.
    
-  We conducted an ablation study simulating noisy data inclusion, demonstrating the model's resilience. 
    
-   We clarified that only publicly available blockchain data was used, ensuring privacy protection. 
    

----------

**Reviewer NnUh:**

-   We expanded the introduction and added references to support the significance of transaction-level anomaly detection. 
    
-   We provided more details about the dataset, including criteria for malicious transaction selection and the rationale for dataset size. The reviewer raised a concern about class imbalance, which we addressed by explaining that malicious transactions were used only for testing.
    
-   We explained the use of different architectures for Ethereum and Solana due to differences in dataset size. 
    
-   We highlighted the depth of our contributions and efforts to ensure the paper moves beyond a technical report. 

----------

In conclusion, we have thoroughly addressed the reviewers’ concerns, with many points clarified and acknowledged during the discussion period. For concerns without explicit responses, we believe our detailed clarifications sufficiently addressed the issues raised. We hope this summary provides a clear overview of our efforts to engage constructively with the reviewers.

Thank you for your time and for overseeing this process.

Best regards,  
Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a deep learning approach for anomaly blockchain transaction detection: BlockFound. It proposed a modularized tokenizer to handle multimodal blockchain-specific elements in the transactions, balancing the information across different modalities. The approach trains a foundation model using a mask language modeling is used to reduce computational costs. A large experimental evaluation is provided.

Strengths:  
- a new method deals with blockchain transactions,  
- multi-modal approach,  
- use of light BERT-like model and masked language learning,   
- comprehensive and promising evaluation real-world data.  

Weaknesses:
- originality limited,   
- definition of anomaly transactions ambiguous,   
- some aspects of the methods should be better discussed and evaluated (use of transactions with top-frequencies,  size of malicious transactions limited, presence of duplicate transactions) 
-generalization and adaptability not well evaluated,  
- privacy issues raised questions.   


During the rebuttal, authors did a strong effort to provide specific answers to the remarks raised by reviewers, they also proposed additional experiments and a revision of the paper.  
During the discussion, the reviewers highlighted the interest of the paper as a unique approach to blockchain transactions and that the authors have addressed some of reviewers' issues.  However, there was a clear consensus for saying that the paper lacks a depth analysis,  a theoretical support and should consider a restructuration.  
While the paper has merits, they agreed that the paper is not ready in the proposed revision.  

Following these elements, I propose rejection.  
Nevertheless, I encourage the authors to improve their work for other venues.