# DExter: Learning and Controlling Performance Expression with Diffusion Models
## 4 Data, Representation, and Processing  
### Input and Target Encodings  
The **performance codec** (p_codec)--our prediction target--was originally proposed in the expressive rendering framework _Basis Mixer_[11], in which four expressive parameters are computed for each note, \(n\), appearing in the score. These parameters of the p_codec encoding the expression controls modify the properties of notes in a MIDI piano performance, thus changing the speed and loudness of the performance with time. Combined with score information, the full expressive performance can be reconstructed in a lossless fashion. We expanded the original Performance Codec v.1.0 [2,11] by defining an additional parameter for sustain-pedal control. The resulting five (note-wise) performance parameters are as follows:  
* Beat period: The ratio of the inter-onset intervals (IOIs) between two consecutive notes of the performance and the score. This parameter is computed for each onset, \(o_{k}\), instead of each note, \(n_{i}\). It is defined as follows: \(x_{bp}(o_{k})=\frac{\mathrm{IOI}_{\text{perf}}(o_{k})}{\mathrm{IOI}_{\text{ learn}}(o_{k})}=\frac{\delta_{k+1}^{p\text{perf}}-\delta_{k}^{p\text{perf}}}{ \delta_{k+1}-o_{k}}\), where \(\delta_{k}^{p\text{perf}}\) is the actual performed onset time, in seconds, corresponding to score onset \(o_{k}\) in beats, calculated as the average onset time of all notes played at score onset position \(o_{k}\). * Velocity: \(x_{\text{rel}}(n_{i})=\frac{\mathrm{vel}(n_{i})}{\mathrm{I}2^{\prime}}\), where vel is the MIDI velocity of a played note. * Timing: \(x_{\text{rim}}(n_{i})=\Delta_{t}(n_{i})=\vartheta^{\text{perf}}(n_{i})-\text{ onset}(n_{i})\), the average onset time of all notes played at the score onset position (used in beat period), minus the performance onset time of \(n_{i}\). The beat period is taken as the 'tempo grid' notion [31]; timing would then refer to the micro-deviation of each note relative to the grid. * Articulation ratio: \(x_{\text{art}}(n_{i})=\frac{\mathrm{d}\mathrm{ur}^{\text{perf}}(n_{i})}{ \mathrm{d}\mathrm{ur}(n_{i})\cdot x_{\text{IP}}(n_{i})}\), which measures the fraction of the expected note duration that is actually played. * Pedal: \(x_{\text{p}\text{det}}(n_{i})=\frac{\mathrm{ped}(n_{i})}{\mathrm{I}2^{\prime}}\), where \(\mathrm{ped}(n_{i})\) is the discrete MIDI pedal value at the onset of \(n_{i}\). Note that pedal encoding is not lossless since changes in the value between note onsets will not be captured. The p_codec is fully invertible in that the full event information from the MIDI file [Pitch, Onset, Duration, and Velocity] can be reconstructed, given the p_codec and score s_codec. The score **codec** (s_codec) represents the musical score, and it is derived from the note array from the _partitura_ package [32]. Aligned with the p_codec at the note level, it contains four score parameters for each note: (notated) Onset, Duration, Pitch, and Voice, resulting in a 2D matrix of dimension \(4\times n\), where n is the number of notes. The score is indispensable for performance conditioning, as it defines the musical content of the piece. The **perceptual features codec** (c_codec), which we use as a steering input for performance generation, is a representation of the so-called _mid-level perceptual features_[4], namely _melodiousness_, _articulation_, _trivhm complexity_, _trivhm stability_, _dissonance_, _tonal stability_, and _'minorness_' (or _mode_). They describe musical qualities that most listeners can easily perceive. Taking a cue from previous research [5,33] showing that these features effectively represent musical factors underlying a wide range of emotions and capture variations in the expressive character between different performances of a piece [34], we incorporated these as the performance-steering inputs. In our scenario, these features are calculated via a previously trained specialized model [5] over the recorded audio performance data of the Vienna4\(\times\)22, (n)ASAP, and ATEPP datasets (see Section 4.4).