## 1 Introduction
They extracted the rotor-stator rub and oil-whirl fault feature using the proposed method. [PERSON] et al. [PERSON] et al. (2018) used VMD to detect defect impulses under variable speeds and loads, and verified the feasibility via analysis of gear transmission dynamics simulated signals. [PERSON] et al. [PERSON] et al. (2018) incorporated bi-dimensional VMD with discriminant diffusion maps to analyze the vibration signals of coal cutters, and detected the rolling bearing cracks. [PERSON] et al. [PERSON] et al. (2018) applied genetic algorithm to optimize the balancing parameter and the number of modes, and combined this improved VMD with 1.5D envelope spectrum to extract compound fault signature of rolling bearings and gearboxes. [PERSON] et al. [PERSON] et al. (2018) decomposed cutting force signals into constituent components via VMD and wavelet packet decomposition, and detected milling chatter based on the energy entropy of constituent components. [PERSON] et al. [PERSON] et al. (2018) proposed an adaptive version of VMD, by optimizing the number of modes and data fidelity constraint with kurtosis as an optimization index via artificial fish swarm algorithm, and they applied the method to rolling bearing fault detection. These studies have demonstrated that VMD can effectively decompose complicated multi-component signals into AM-FM components, and suppress noise interferences, thus enhancing fault features manifested by modulations. Therefore, VMD provides an effective approach to demodulation analysis of planetary gearbox vibration signals. In this paper, two major contributions are made. Firstly, considering the complex AM-FM features of planetary gearbox vibration signals, and exploiting the merits of VMD in AM-FM component decomposition of complex modulated signals, we propose a joint amplitude and frequency demodulation analysis idea for planetary gearbox fault diagnosis, thereby to overcome the difficulty with conventional Fourier spectrum analysis due to sideband complexity, and reveal the gear fault feature in a more effective way. Secondly, in order for effective implementation of the proposed demodulation analysis, we propose solutions to address the issues existing with application of VMD in planetary gearbox vibration signal demodulation analysis. These solutions can also be extended to applications of VMD in other signal analysis tasks. (1) The number of intrinsic mode functions (IMFs) to be separated is a key parameter for VMD. We propose a criterion to determine the number according to the spectral characteristics of planetary gearbox vibration signals. (2) Among those IMFs obtained from VMD, which IMF contains the major information about and is sensitive to gear fault is also an important issue. We propose a criterion to select the sensitive IMF for further demodulation analysis, based on the modulation characteristics of gear vibration signals and the wavelet packet like decomposition property of VMD. (3) Accurate instantaneous frequency estimation is necessary for effective frequency demodulation analysis, but conventional Hilbert transform (HT) based approach is subject to the constraint imposed by Bedrosian and Nuttall theorem [37]. We preprocess the IMFs with empirical AM-FM decomposition by exploiting its capability to separate the carrier signal from amplitude envelope of an AM-FM signal, to satisfy the non-overlap requirement of amplitude envelope and carrier signal spectra by Bedrosian and Nuttall theorem. Hereafter, this paper is organized as follows. Sections 2 and 3 introduce the principle of VMD and empirical AM-FM decomposition, respectively. Section 4 proposes a joint amplitude and frequency demodulation analysis method. Section 5 illustrates the method using a simulated planetary gearbox signal. Section 6 further validates the method by analyzing the lab experimental datasets of a planetary gearbox. Section 7 draws conclusions.