# Grey-Box Fuzzing Based on Reinforcement Learning for XSS Vulnerabilities
## 1 Introduction
On the dataset provided by the authors, the accuracy rate of XSS vulnerability detection reaches 99.47%. However, detection based on deep learning is generally tricky for real-world web applications. This is because most web pages are dynamically generated, and it takes a lot of human effort to convert them into a format that deep learning models can recognize. In general, fuzzing-based XSS vulnerability detection is practical and can find vulnerabilities in real-world web applications. However, this method still has problems, such as low payload generation efficiency. We consider that there are three challenges for fuzzing-based methods: (1) diverse web frameworks. A web framework brings convenience to developers but makes web applications more complex, making the analysis of web application code and configuration inaccurate; (2) input point identification is incorrect. Most fuzzing-based methods use crawlers to discover potential input points. However, this cannot guarantee that all input points are covered; (3) inefficient payload generation. Due to the increasing scale of web applications, traditional payload generation strategies (such as random fuzzing) cannot generate test cases in good time. In this paper, we overcome the above challenges and propose a reinforcement learning-based grey-box fuzzing method for Java web applications. Our approach is inspired by [1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13], which propose a Q-learning model-based SQL injection vulnerability detection. We further extend [1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13] to make it possible to detect XSS vulnerabilities. On the one hand, we use static analysis to scan every component of the Java web application for potential input points. On the other hand, we experimented with the performance of three reinforcement learning models on XSS vulnerability detection. Compared with traditional payload generation, we detect vulnerabilities in a shorter time. Overall, the main contributions of this paper are as follows:  
1. We use static analysis to identify input points of Java web applications, including Java code (supporting Java Servlet and Spring framework), configuration files, and HTML code. Almost all potential input points of the web application could be covered. 2. We propose an XSS payload generation method based on reinforcement learning. The reinforcement learning model's state, action, and reward function are defined in the XSS vulnerability detection scenario. We validate payload generation by DQN, DDQN, and Policy Gradient model. 3. To evaluate the effectiveness of the proposed method, we implement it and design systematic experiments. On the one hand, we compare the efficacy of different reinforcement learning models in XSS vulnerability detection. On the other hand, we compare the performance of the proposed method with four state-of-the-art web scanners. Experimental results show that our method is more effective. The remainder of this paper is organized as follows. Section 2 introduces the related work about XSS vulnerability detection and reinforcement learning for cyber security. Research progress is discussed in detail. Section 3 presents the background of XSS vulnerability and reinforcement learning to help readers quickly understand the two fields. In Section 4, the composition and principles of the proposed method are introduced in detail. To evaluate our method, we conduct comprehensive experiments in Section 5. We propose three standard research questions and demonstrate the effectiveness of our tool by answering them. Section 6 summarizes the work in this paper and introduces future work.