## 1 Introduction
According to Reference [28], this paper will further explore the influence of the relative position of two faulty teeth on the dynamic characteristics of gears. The accuracy of the dynamic model needs to be verified by the experimental signal. However, the experimental signal has the background noise caused by the sensor's location (e.g., the bearing pedestal or the gearbox housing), which will affect the results of dynamic model accuracy verification. Moreover, the fault features hidden in the experimental signal are also susceptible to the background noise and become weaker, which increase the difficulty of fault feature analysis. It is of great significance for dynamic model validation and fault feature analysis to eliminate the background noise in the experimental signal. To tackle the complex background, several signal processing methods such as Wavelet Transform (WT) [29; 30], Empirical Mode Decomposition (EMD) [31; 32], and Local Mean Decomposition (LMD) [33; 34] have been applied. However, the treatment effect of these methods is often not satisfactory because of the inherent limitations. The WT method needs to empirically choose wavelet basis function. The EMD method decomposes any signal into a set of intrinsic mode functions (IMFs), but its efficiency is restricted by the problems of end effects and mode mixing. The LMD method is confined by the decomposition level. In recent years, a novel adaptive signal processingmethod called Variational Mode Decomposition (VMD) is presented [35]. By determining the relevant bands of the fault feature and dividing the signal into several IMFs, the VMD can effectively avoid the end effects and mode mixing [36; 37]. In addition, [PERSON] et al. [38] and [PERSON] et al. [39] proved that the VMD performs better than the EMD. With the merits, many novel signal processing methods around the VMD have been proposed [40; 41; 42; 43]. However, the performance effectiveness of the VMD is mainly affected by the penalty factor \(\hat{a}\) and the decomposed mode number \(K\), which are obtained empirically at present. Ant lion optimization (ALO) is a novel intelligent algorithm, which mainly solves the optimization of the parameters by imitating the hunting mechanism of antions [44]. Compared with other intelligent algorithms such as a genetic algorithm (GA) and particle swarm optimization (PSO), the ALO has the merit of fewer control variables and high convergence precision in the classical optimization problem [45; 46]. With the merit, we adopt the ALO to search the optimal parameters (i.e., the penalty factor \(\hat{a}\) and the decomposed mode number \(K\)) in the VMD, which ensures the decomposition performance of the VMD. Generally, gear system dynamic characteristics and fault feature analysis that considers the double-teeth spalling fault have not been well studied in existing research. To make this gap, a new dynamic model of gear systems considering the double-teeth spalling fault is established in this paper. This dynamic model is validated by various experimental tests, and could supply more realistic dynamic environment for extraction of a gear double-teeth fault feature. Then, the dynamic characteristics considering the effect of the number of spalling teeth, and the relative position of two faulty teeth are analyzed. In addition, to ensure the accuracy of dynamic model verification results and reduce the difficulty of fault feature analysis, a novel parameters-adaptive VMD based on the ALO is proposed to eliminate the background noise in the experimental signal. The optimal parameters K and a of the VMD is obtained by the ALO. Moreover, the raw signal acquired from gear systems is decomposed into several IMFs by using the ALO-VMD method, and the IMFs with effective weight kurtosis (EWK) are summed as a reconstructed signal. Then, envelope spectrum analysis is applied to filter the reconstructed signal and carried out to extract the features of the faults.