## 1 Introduction  
With the development of information technology, the network environment is increasingly complex, and network security threats are becoming increasingly severe. How to protect computer systems from unauthorized access or attack is an essential issue [1]. As a proactive method, the penetration test evaluates network vulnerability by discovering potential threat paths of target networks from the perspective of a penetration tester [2]. However, with the growth of the network information system at a large scale and complexity, it takes a lot of time to complete a penetration test, and there are a lot of repetitive actions in the process, which results in high costs [3]. Therefore, it is extremely urgent to examine how to carry out penetration tests autonomously and intelligently. Automated penetration tests have been proposed in recent years. Early studies, such as attack tree, attack graph, and planning field programming language (PDDL), are representative. These methods plan the attack path through a formal representation of the target network configuration information and the analysis of state transition [2; 3; 4]. However, they all need to know the target network information in advance and could not model the uncertainty of the real penetration test well. With the development of the field of artificial intelligence, reinforcement learning methods have obvious advantages in sequential decision-making problems. The Markov decision process (MDP) formally describes the environment in reinforcement learning (RL), which underlies the intelligent penetration test. It is necessary to adopt RL to construct a dynamic environment to train the attack agent for the intelligent decision of exploit actions. The environment could provide appropriate feedback and incentive for the agent to drive the agent to explore and exploit the environment [5]. Penetration tests in the real world are a complex process. In order to train the penetration tester agent better to adapt to the complex penetration test scenarios, it is significant to construct a highly simulated environment of RL. At present, simulation environments of the penetration test based on RL have been proposed, such as Nasim [6], CybORG [7], CyberBattleSim [8] and CyGIL [9]. The simulation environments mentioned above are widely used in the existing research, and they are stable and extensible. However, there are still some problems to be solved. On the one hand, they are not fully simulating all attack actions of penetration tests in the real world. In addition, with the development of modern security systems, it becomes more and more difficult for penetration testers to perform a penetration test on the target network [10], and human beings cannot complete tasks according to fixed instructions like a machine. It is inevitable that mistakes will occur in the process of performing tasks. Social engineering is a way of grasping human defects and exploiting the target. The method is a supplementary and important way for testers to conduct penetration tests [11]. Therefore, it is significant to extend the penetration tester's actions from the perspective of social engineering. On the other hand, the construction of a simulation module in the existing simulation environment is based on the traditional network graph model, which only includes the basic elements and connection relations of the network and does not incorporate the security attributes and elements related to the penetration test. Therefore, it is necessary to improve the exited network graph model and integrate it into security-related attributes and elements in the penetration test. In conclusion, the following challenges need to be stressed:  
* Challenge 1: It is inevitable that mistakes will occur in manual penetration tests. Social engineering is a useful way to exploit human defects. However, the existing simulation environment does not incorporate social engineering.