# Adaptive Filtering for Multi-Track Audio Based on Time-Frequency Masking Detection
## 2 Evaluation of Masking and Filtering  
Our method first calculates the magnitude of the short-time Fourier transforms \(T\) of each track:  
\[T(t,f)=|F\{T_{r}(t_{0})*w(t_{0}-t)\}| \tag{1}\]  
where \(T_{r}\) is the raw track, \(t_{0}\) the temporal dimension, \(w\) the window function, and \(t\) the time offset of the window function. The purpose of the window function is to suppress the signal outside of the region of interest to provide resolution in the time domain. In this case, we have used a Hann window [29] of 180 ms in length and spaced by 90 ms. This is a compromise between temporal and spectral resolutions, which enables resolving individual notes (see Figure 1a,b). We propose Equation (2) to obtain an array of values that identify where in time and frequency masking is occurring, at which severity and in which _polarity_ (i.e., differentiating masker and _maskee_). We define a masking matrix \(M^{\prime}(f,t)\) by:  
\[M^{\prime}_{12}(t,f)=\frac{(T_{1}(t,f)*T_{2}(t,f))(T_{1}(t,f)-T_{2}(t,f))}{( mean(T_{1}(t,f))+mean(T_{2}(t,f)))^{3}} \tag{2}\]  
where \(T_{1}\) and \(T_{2}\) are the magnitude of short-time Fourier transforms of track 1 and track 2, respectively. The expression \(mean()\) indicates a 2D average (scalar). This process creates a matrix of masking \(M^{\prime}_{12}(t,f)\) between specific tracks, 1 and 2 (see Figure 1c). The equation consists of three parts:  
\((T_{1}(t,f)*T_{2}(t,f))\)--this term evaluates how relevant it is to calculate masking at this specific time and frequency. For instance, if the amplitude of one track is close to zero, then masking would be small. \((T_{1}(t,f)-T_{2}(t,f))\)--this term evaluates the relative magnitude and polarity of masking. Positive values for \(T_{1}\) indicate this is the masker while negative indicates it is the mas see. This is an important term to differentiate the masker from the maskee at different times. \((mean(T_{1}(t,f))+mean(T_{2}(t,f)))^{3}\)--this is a normalisation term to ensure scale invariance that the same results will be provided if the same non-zero dB gain is applied to both tracks. The two-dimensional masking matrix \(M^{\prime}(f,t)\) is then reduced into a one-dimensional form where all the temporal information is summed while the frequency space is preserved:  
\[M^{\prime}_{12}(f)=\sum_{t=0}^{t=N_{t}}M^{\prime}_{12}(t,f) \tag{3}\]  
where \(t\) is the time position of the window function and \(N_{t}\) is the total number of time samples (see Figure 2a). Even though the temporal information is compressed, this only occurs after the calculation of masking takes place. Therefore, the frequencies that contribute to \(M^{\prime}(f)\) are only those that overlap in both time and frequency. This is key to preventing filtering elements of the signals whose components do not exist at the same time in the tracks. After the data are reduced to a single dimension, the positive and negative parts, which correspond to the masking of T1 over T2 (T1 is the masker) and vice versa (T1 is the masquee), are separated. Therefore, \(M^{\prime}_{12}(f)\) becomes:  
\[M_{12}(f)= \begin{cases}M^{\prime}_{12}(f),&\text{if }M^{\prime}_{12}(f)>0\\ 0,&\text{otherwise}\end{cases}  \tag{4}\]  
\[M_{21}(f)= \begin{cases}|M^{\prime}_{21}(f)|,&\text{if }M^{\prime}_{12}(f)<0\\ 0,&\text{otherwise}\end{cases}  \tag{5}\]  
where \(M_{12}(f)\) and \(M_{21}(f)\) are the contributions to masking from the masker and the maskee, respectively (see Figure 2b,c). After calculating all contributions to masking from each track to the others (for an arbitrary number of tracks), these are summed and normalised:  
Figure 1: Calculation of masking in the temporal and spectral dimensions using short-time Fourier transforms. The time–frequency spectrum of tracks 1 ((a), bass) and 2 ((b), guitar) was obtained using short-time Fourier transforms. (c) Masking between track 1 and track 2 as given by Equation (2). The positive values in (c) indicate the masking of track 1 over track 2 while the negative values in (c) indicate the masking of track 2 over track 1. \[