# Musical Mix Clarity Prediction Using Decomposition and Perceptual Masking Thresholds
## 2 Proposed Model
### Transient, Steady-State, and Residual Separation Stage  
When considering a spectrogram representation, the TSR components are characterised as follows:  
* Transient components: Spectrally broadband and temporally transient bursts of energy which form strong vertical beams on the magnitude spectrogram [28], and unpredictable changes in phase between consecutive frames [29]. * Steady-state components: Slowly evolving harmonic partials with predictably evolving phase between consecutive frames [29], forming strong horizontal beams on the magnitude spectrogram [28]. * Residual components: Cannot be classified as transient or steady-state. These are noise-like components of the signal, described as the 'texture' of the sound [30], which are generally broadband and stochastic in terms of magnitude and phase. The TSR model gives a more general description of signal components that do not require specific instrument classification. Following the suggested negative impact of noise-like signals on mix clarity [20; 21], greater masking of the residual (\(R\)) component would indicate a less audible residual component and therefore a potentially higher perceived mix clarity. Additionally, and perhaps more importantly, less masking of the transient and steady-state (TSS) components would indicate greater audibility of the percussive (rhythmic) and harmonic (pitched) parts of the signal. Transient onsets have also been linked to instrument identification [31], suggesting potentially greater mix clarity perception where TSS components are less masked. Considering the HPM system [26], masked residual  
Figure 1: Block diagram of the proposed transient, steady-state and residual based masking metric. energy bears some resemblance to the idea of masked non-dominant signals, whose presence may be unnecessary or even detrimental to the perceived clarity of the mixed signal. Moreover, the importance of TSS and R components can be thought of as a simple hierarchy, where the presence of the TSS components take priority over R components. In this regard, the residual energy may not necessarily be detrimental to the perception of clarity where it is below a level which would mask the TSS energy. On the contrary, as the residual information contains cues which dictate the texture and timbre of instruments, its presence may be beneficial to a mix where it is properly balanced. The TSR separation stage adopts a median filter based approach [30], chosen for its efficiency in the perceptual measure of punch [1; 32] and its relative simplicity. For this, the magnitude spectrogram (\(S(k,\,t)\)) of the input signal is taken using a short time Fourier transform (STFT), where \(k\) indicates bin index, and \(t\) indicates frame index. In this case, the STFT was performed on a 2048 sample window, multiplied with a Hanning window function, and a 50% overlap between frames. The magnitude spectrogram is then filtered with a sliding 17 th order median filter. Filtering is performed vertically across the bins, yielding the transient emphasised spectrogram (\(S_{T}\)), and horizontally across the frames yielding the steady-state emphasised spectrogram (\(S_{SS}\)). Binary masks (\(BM\)) were then defined for the transient and steady-state components as:  
\[ \begin{split} BM_{T}(k,t)=\left(\frac{S_{T}(k,t)}{S_{SS}(k,t)+ \varepsilon}\right)>\beta_{T}\\ BM_{SS}(k,t)=\left(\frac{S_{SS}(k,t)}{S_{T}(k,t)+\varepsilon} \right)>\beta_{SS}\end{split}  \tag{1}\]  
where \(\varepsilon\) is a small offset to avoid division by 0, \(\beta_{T}\) is the transient threshold set at 1.75, and \(\beta_{SS}\) is the steady-state threshold set at 1. These parameter values were chosen based on previous works, namely the perceptual punch meter [1; 32]. The residual component mask is determined to be any bins which do not appear in the transient or steady-state masks, as:  
\[BM_{R}(k,t)=1-(BM_{T}(k,t)\mid BM_{SS}(k,t)) \tag{2}\]  
Using the masks, the spectrograms of the transient (\(T\)), steady-state (\(S\)) and residual (\(R\)) components can then be extracted:  
\[