###### Abstract
DRL is an RL algorithm that uses a neural network as a parameter structure for optimization [15], which means that DRL is a potent data-driven technique that makes use of the agent's interaction with the dynamic environment to learn the best course of action. On the basis of the action space, DRL settlement can be separated into continuous and discrete control problems [16]. For discrete issues, the traditional deep Q-learning (DQN) approach is appropriate [17], while deep deterministic policy gradient (DDPG) [18] and TD3 are appropriate for continuous problems. The study in [19] investigated an RL-based adaptive robustness parameter tuning approach for the VSG; however, due to the defect of DQN being used to deal with discrete action problems, the data makes no attempt to engage with the current discussion on continuous action. Although, a control strategy based on DDPG was proposed in [20], allowing for flexible adjustment of the damping and inertia coefficient in response to changing working conditions, the issue of failure brought on by an inflated Q value was not considered. In the above-mentioned literature related to VSG control strategies, both the improved control structure and intelligent thought methods were given. Few of them considered the TD3 algorithm with regard to the performance of suffering from the active load, frequency, and voltage change, To fill the research gaps of DRL applications in such areas, a TD3-based MMC-VSG structure was developed. In this study, a DRL learning module is employed to replace the VSG module in the control system, which means that the topology of the control system can be much simpler. Moreover, the pre-trained parameter of VSG can be adjusted according to the system change, which will help the system fit into dynamic circumstances. In view of this, this paper mainly studies the effectiveness and robustness of the DRL learning module and its typical usage in the TD3-VSG control system. The remaining parts of the study are listed as follows: First, the reasons that cause the contradiction in MMC-VSG's dynamic and steady-state are analyzed. A mathematical model of MMC-VSG and its topology structure is presented, and the VSG control strategy is examined. Secondly, the study focuses on the damping and inertia characteristics affecting the VSG control; the contradiction between J (the moment of inertia) and D (damping coefficient) compels the system to work with the relative \"best\" working conditions to reconcile transient and steady states. Thus, a DRL-based model, which is proposed in the following section, can separate the dynamic steady state in VSG control. Thirdly, the Markov decision process (MDP) and the TD3 algorithm are analyzed. The TD3 method is a DRL system developed in light of both the decision-making and perceptual capabilities of deep learning. According to the analysis, a TD3-based MMC-VSG control strategy for replacing the conventional VSG submodule is illustrated. Lastly, based on the prototype, the TD3-based MMC-VSG prototype is built using the DRL toolbox in MATLAB/Simulink software to verify the validity of the raising strategy. Furthermore, a comparison model of DDPG-VSG, fixed-VSG, and vector MMC is also built to demonstrate the effectiveness and superiority.