## 2 Related Works  
Extensive research on formal methods has been conducted on the detection of vulnerabilities in smart contracts in earlier times. Traditional methods, distinguished by the software testing techniques utilized, can be classified into four categories: symbolic execution, formal verification, fuzz testing and static analysis [12]. Symbolic execution was first used in automated analysis tool such as Oyente [13] and Mythril [14] for Ethereum smart contracts, where it simulates all possible execution paths on contract codes for detecting potential vulnerabilities but may face difficulty when analyzing complex contracts. The formal verification tool Security [15] and static analysis tools such as SmartCheck [16] and Slither [17] improve the efficacy of addressing complex issues by adopting a specialized domain-specific representation to describe compliance and violation patterns, thereby facilitating the identification of potential security flaws. However, it may overlook new vulnerabilities that do not conform to known patterns. Numerous dynamic analysis tools, including ReDefender [18] and ConFuzzius [19], have gained popularity as the fuzz testing technique develops. These tools generate fuzzing inputs and verify vulnerabilities through dynamic data dependency analysis, but their detection scope is restricted to a narrow window of vulnerability types. Traditional tools predominantly depend on predefined hard logic rules established by experts for detecting vulnerabilities. However, as smart contract technology advances, these tools are becoming insufficient to meet contemporary requirements due to the following limitations [20]. * The complexity of smart contract structures and functionalities is increasing. As the variety of vulnerabilities in smart contracts continues to grow, expert-defined rules based on existing vulnerability definitions struggle to keep pace with these updates. * Smart contracts may demonstrate varied behaviors at runtime depending on different conditions, complicating the process of vulnerability verification and rendering the analysis highly resource- and time-consuming due to their dynamic and conditional nature. In recent years, machine learning-based approaches for detecting vulnerabilities in smart contracts have garnered extensive attention and research, as more accurate and efficient results are achieved during smart contract security analysis. Within this field, supervised learning techniques have in particular made significant progress and are widely applied in real-world scenarios, including the web service tool SmartEmbed [21] for detecting repetitive code and clone-related bugs and the support vector machine embedded scanning system SCScan [22], a tree-based approach using an improved CatBoost algorithm for early detection of Ponzi Smart Contracts on Ethereum [23]. More deep learning approaches have emerged since 2020. The bidirectional Long Short-Term Memory networks enhance detection accuracy for reentrancy vulnerabilities and reduce the occurrence of false positives by incorporating contract snippet representations and attention mechanisms [24,25]. And convolutional neural network and graph neural network architectures are used to detect vulnerabilities in smart contracts and achieve over 80% accuracy by maintaining the semantic and contextual information under abstract syntax tree representations [26,27]. There also emerges specialized bidirectional encoder representations from transformer models for smart contract vulnerability detection to mitigate the challenge of insufficient labeled data [28,29]. While supervised learning techniques have yielded notable successes in the detection of smart contract vulnerabilities, reinforcement learning approaches have received relatively less attention and exploration in this field. [PERSON] et al. 's [30] review paper proposed a deep reinforcement learning with multi-agent fuzzing for exploit generation on cross-contract, and [PERSON] et al.