## 1 Introduction
In contrast, reinforcement learning provides an alternative strategy where agents can learn optimal detection policies through interactions, even in the absence of exhaustive labels or prior definitions of vulnerability patterns. Beyond smart grids, spatio-temporal prediction networks have been applied in various security-critical domains. [PERSON] [9] proposed a spatio-temporal graph neural network (STGNN) for fraud detection in blockchain, combining graph convolution for transaction structure and gated recurrent units to model temporal behavior. Similarly, [PERSON] et al. [10] introduced TENET, a lightweight Temporal CNN with attention, for detecting cyber intrusions in automotive systems, demonstrating high efficiency and accuracy under resource constraints. In the industrial context, [PERSON] et al. [11] utilized a multi-scale dynamic graph neural network to detect anomalies in multivariate sensor data, capturing both its hierarchical structure and temporal evolution. These approaches underscore the value of modeling both execution flow and time-series dynamics to identify malicious behaviors. Inspired by these works, our study applied similar principles to the smart contract setting, while shifting from supervised learning to reinforcement learning, allowing agents to adaptively explore and identify vulnerabilities without requiring static labels or exhaustive expert knowledge. Machine learning techniques have proven to be an effective approach for detecting security vulnerabilities in smart contracts. While traditional methods like static analysis and symbolic execution are used to identify issues such as reentrancy, integer overflows, and timestamp dependencies, these automated techniques are limited to detecting known vulnerabilities, are often time-intensive, and depend heavily on predefined rules from experts. As a result, machine learning and deep learning approaches have been developed to minimize the need for heavy feature engineering. Despite the growing success of supervised learning-based detection methods, the exploration of reinforcement learning in smart contract vulnerability detection remains relatively underdeveloped in the existing literature. To address this gap, this paper applied reinforcement learning techniques to the process of smart contract vulnerability detection and conducting an empirical analysis with a substantial dataset. The main contributions of this paper are as follows:  
1. We designed and implemented a reinforcement learning architecture for detecting and classifying smart contract vulnerabilities and showed its viability through a comparison analysis. 2. Based on the characteristics of smart contract bytecode, we proposed CFGs, a graph-base method for input representations to capture richer semantic information within the bytecode. 3. We proposed a machine learning approach for improving blockchain security and future reinforcement learning applications. The rest of the paper is organized as follows. Section 2 provides a comprehensive review of the research work related to this paper. In Section 3, we elaborate the data work in the analysis and the methodology for building feature representations and model architectures. Section 4 examines and discusses the experimental results, and Section 5 summarizes and highlights possible future research work. The code for the paper is presented in the Supplementary Materials section.