# DExter: Learning and Controlling Performance Expression with Diffusion Models
## 3 Methodology
### Diffusion Framework  
We frame the expression-rendering problem as the task of learning a continuous space of performance expression parameters, as shown in Figure 1. Diffusion models [27] consist of two processes: (i) a forward process that transforms each data sample into a standard Gaussian noisy sample, step by step, with a predefined noise schedule; and (ii) a reverse process in which the model learns to denoise pure-noise inputs gradually, generating samples from the learned training data distribution. In effect, our model aims to convert Gaussian noise \(x_{t}\) into a posterior performance codec, \(\hat{x}_{0}\), conditioned on a score codec, \(c_{s}:=\) s_codec, and a perceptual features codec, \(c_{c}:=\) c_codec. The **diffusion forward pass**\(q(x_{t}|x_{0})\) produces a probability distribution of progressively noisier versions of the performance codec, conditioned on the original. With the noise \(\epsilon\in\mathcal{N}(\mathbf{0},\mathbf{I})\) sampled from a standard Gaussian distribution, we blend it with the input sample \(x_{0}\), using \(\beta\) as a scaling factor intended to ultimately achieve a zero mean and unit variance of the fully noised result. Specifically, the sampling process applies a linear noise schedule with \(\beta\in[0.0001,0.2]\). As we would like to perform multiple steps simultaneously, reparameterization is applied to derive a closed-form equation, given that \(\alpha_{t}=1-\beta_{t}\) and \(\bar{\alpha}=\prod_{s=1}^{t}\alpha_{s}\). \[x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon \tag{1}\]  
During **training**, model \(f_{\theta}(x_{t},t,c_{s},c_{c})\) learns to predict the injected noise \(\hat{e}_{\theta}\), given a random timestep, \(t\), and its noised codec version, \(x_{t}\), calculated in the forward pass. \(t\) is sampled from \([1,T]\); we used \(T=1000\) in our experiments. Then, we use the predicted noise \(\hat{e}_{\theta}\) to reconstruct the predicted initial codec \(\hat{x}_{0}\) by inverting Equation (1). The parameter \(T\), or the time interval limit, significantly influences the granularity and smoothness of the denoising process. A higher \(T\) allows for a more gradual denoising process, potentially capturing more subtle nuances in the data as it iteratively approximates the clean state. The training objective combines noise estimation and codec reconstruction, as shown in Equation (2). Although theoretically, noise prediction alone could suffice to train the model, the empirical results demonstrate that adding a constraint to the reconstructed codec significantly improves performance. We use a weighting factor, \(h=0.2\), derived empirically, to balance the emphasis between accurate noise prediction and fidelity in codec reconstruction, optimizing both the precision of the generated output and the stability of the learning process. Figure 1: Training (**left**) and inference (**right**) phases of the diffusion framework. Training starts with p_codec and corrupts the \(x_{0}\) by injecting noise; the UNet model takes in the corrupted conditions of \(x_{t}\), \(c_{s}\) and \(c_{c}\), to predict the injected noise, which is then used to reconstruct \(\hat{x}_{0}\). The loss is calculated for both noise prediction and p_codec reconstruction. The inference process (**right**) starts with a random sample from \(\mathcal{N}(\mathbf{0},\mathbf{I})\); the model iteratively predicts the noise and reconstructs \(\hat{x}_{0}\), conditioned on the same \(c_{s}\) and \(c_{c}\). Alternatively, for transferal, we initialize the process from another performance, \(y_{0}\), corrupting it for \(t_{0}\) steps and denoising for the remaining \(T-t_{0}\) steps.