# MPFM-VC: A Voice Conversion Algorithm Based on Multi-Dimensional Perception Flow Matching
## 4 Methods
### Multi-Dimensional Perception Network
Specifically, the embedding method of each feature is as follows:  
* Energy \(I_{\textit{energy}}\) is calculated as the root-mean-square (RMS) energy of each frame in the speech signal. The first perception block utilizes the energy embedding, \(E_{\textit{energy}}\), to keep the energy of input and output stable. * Pitch information \(I_{\textit{pitch}}\) is extracted by using the pre-trained neural pitch estimation model RMVPE [42], which directly derives pitch features from raw audio. The second block employs the pitch embedding, \(E_{\textit{pitch}}\), to maintain high reduction in and coherence of the output pitch. * Prosodic features \(I_{\textit{prosody}}\) are extracted by using the neural prosody encoder HuBERT-Soft [43], which captures rhythm, intonation, and other prosodic cues in speech. The third block uses the prosody embedding, \(E_{\textit{prosody}}\), to control the stress and opening of the output speech. Figure 4: The architecture of the multi-dimensional feature perception network. * Speaker identity features \(I_{spk}\) are obtained by using the pre-trained speaker verification model Camplus [44]. The final block integrates speaker characteristics by using the speaker embedding, \(E_{spk}\), to captures speaker-dependent characteristics. These encoded features are fused through a one-dimensional convolutional layer and are ultimately used to predict the vector field, \(v(z_{t},t)\). The core design principle behind this structure is that the closer to the output layer a block is, the stronger its sensitivity to conditional information is. Therefore, in acoustic modeling tasks, the speaker's timbre should be assigned greater importance, while energy-related information can be considered relatively less influential. Nonetheless, each feature-awareness block is capable of adaptively learning and adjusting its sensitivity to various types of conditioning through dynamic parameterization, enabling the model to flexibly control the contribution of each feature and optimize feature fusion strategies based on task-specific requirements. Furthermore, we adopted the Classifier-Free Diffusion Guidance [45] (CFG) approach, which allowed manual adjustment of the contribution of each condition during inference. Traditionally, as shown in Equation (6), the guidance vector field is predicted using a weighted combination of the full condition matrix and its fully masked counterpart. In this work, we introduced an additional term containing only the prosody matrix, as shown in Equation (7), to enable fine-grained control over prosodic features such as stress and openness in the synthesized speech. In this paper, our base inference method did not use CFG, and further prosodic CFG are discussed in ablation experiments. \[\vartheta_{t}(z_{t},c)=\omega_{1}\cdot v_{t}(z_{t},c)+\omega_{2}\cdot v_{t}(z_{ t},\varnothing) \tag{6}\]  
\[\vartheta_{t}(z_{t},c)=\omega_{1}\cdot v_{t}(z_{t},c)+\omega_{2}\cdot v_{t}(z_ {t},I_{prosody},\varnothing) \tag{7}\]  
where \(c\) denotes the conditional state, \(\varnothing\) denotes the unconditional state, \(\omega_{t}\) denotes weight coefficients, and the sum of all the weight coefficients is 1. In conventional flow-matching models, the vector field, \(v_{t}(z)\), is typically trained uniformly across all time steps. However, in practice, predictions at intermediate time steps tend to be more challenging. Specifically, when \(t\) approaches 0, the optimal prediction tends to align with the mean of the target distribution (\(p_{1}\)), while for \(t\) close to 1, it tends to align with the mean of the prior distribution (\(p_{0}\)). In contrast, predictions around the midpoint (\(t\approx 0.5\)) are often more ambiguous and unstable due to increased distributional uncertainty. To more accurately learn the ground-truth vector field, \(u_{t}(z)=z_{1}-z_{0}\), we introduced a log-normal weighting scheme in the time dimension to reweight the loss function within the optimal transport-based flow-matching framework.