# A Review of Time-Scale Modification of Music Signals +
## 6 TSM Based on Harmonic-Percussive Separation
### Harmonic-Percussive Separation  
The goal of harmonic-percussive separation (HPS) is to decompose a given audio signal \(x\) into a signal \(x_{\text{h}}\) consisting of all harmonic sound components and a signal \(x_{\text{p}}\) consisting of all percussive sound components. While there exist numerous approaches for this task [30; 31; 32; 33; 34; 35], a particularly simple and effective method was proposed by [PERSON] in [23], which we review in the following. To illustrate the procedure, let us have a look at Figure 13. The input is an audio signal as shown in Figure 13a. Here, we again revert to our example of a tone played on the violin, superimposed with a single click of castanets. The first step is to compute the STFT \(X\) of the given audio signal \(x\) as defined in Equation (15). A critical observation is that, in the spectrogram \(Y=|X|\), harmonic sounds form structures in the time direction, while percussive sounds yield structures in the frequency direction. In the spectrogram shown in Figure 13b we can see horizontal lines, reflecting the harmonic sound of the violin, as well as a single vertical line in the middle of the spectrogram, stemming from a click of the castanets. By applying a median filter to \(Y\)--once horizontally and once vertically--we get a horizontally enhanced spectrogram \(\tilde{Y}_{\text{h}}\) and a vertically enhanced spectrogram \(\tilde{Y}_{\text{p}}\):  
\[\tilde{Y}_{\text{h}}(m,k) = \text{median}\left(Y(m-\ell_{\text{h}},k),\ldots,Y(m+\ell_{\text {h}},k)\right) \tag{28}\] \[\tilde{Y}_{\text{p}}(m,k) = \text{median}\left(Y(m,k-\ell_{\text{p}}),\ldots,Y(m,k+\ell_{ \text{p}})\right) \tag{29}\]  
for \(\ell_{\text{h}}\), \(\ell_{\text{p}}\in\mathbb{N}\), where \(2\ell_{\text{h}}+1\) and \(2\ell_{\text{p}}+1\) are the lengths of the median filters, respectively (Figure 13c). We assume a time-frequency bin of the original STFT \(X(m,k)\) to be part of the harmonic component if \(\tilde{Y}_{\text{h}}(m,k)>\tilde{Y}_{\text{p}}(m,k)\) and of the percussive component if \(\tilde{Y}_{\text{p}}(m,k)\geq\tilde{Y}_{\text{h}}(m,k)\). Using this principle, we can define binary masks \(\mathcal{M}_{\text{h}}\) and \(\mathcal{M}_{\text{p}}\) for the harmonic and the percussive components (Figure 13d):  
\[\mathcal{M}_{\text{h}}(m,k)=\begin{cases}1,&\text{if }\tilde{Y}_{\text{h}}(m,k)> \tilde{Y}_{\text{p}}(m,k),\\ 0,&\text{otherwise},\end{cases} \tag{30}\]  
\[\mathcal{M}_{\text{p}}(m,k)=\begin{cases}1,&\text{if }\tilde{Y}_{\text{p}}(m,k) \geq\tilde{Y}_{\text{h}}(m,k),\\ 0,&\text{otherwise}.\end{cases} \tag{31}\]  
Figure 12: Transient preservation in TSM based on harmonic-percussive separation (HPS). The input signal \(x\) is the same as in Figure 10 and is stretched by a factor of \(\alpha=1.8\) using TSM based on HPS.  
Applying these masks to the original STFT \(X\) yields modified STFTs corresponding to the harmonic and percussive components (Figure 13e):  
\[X_{\mathrm{h}}(m,k) = X(m,k)\ \mathcal{M}_{\mathrm{h}}(m,k)\, \tag{32}\] \[X_{\mathrm{P}}(m,k) = X(m,k)\ \mathcal{M}_{\mathrm{P}}(m,k). \tag{33}\]  
These modified STFTs can then be transformed back to the time-domain by using the reconstruction strategy discussed in Section 5.2. This yields the desired signals \(x_{\mathrm{h}}\) and \(x_{\mathrm{P}}\), respectively. As we can see in Figure 13f, the harmonic component \(x_{\mathrm{h}}\) contains the tone played by the violin, while the percussive component \(x_{\mathrm{P}}\) shows the waveform of the castanets' click.  
Figure 13: The HPS procedure presented in [23]. (**a**) Input audio signal \(x\); (**b**) STFT \(X\) of \(x\); (**c**) Horizontally and vertically enhanced spectrograms \(\bar{Y}_{\mathrm{h}}\) and \(\bar{Y}_{\mathrm{P}}\); (**d**) Binary masks \(\mathcal{M}_{\mathrm{h}}\) and \(\mathcal{M}_{\mathrm{P}}\); (**e**) Spectrograms of the harmonic and the percussive component \(X_{\mathrm{h}}\) and \(X_{\mathrm{P}}\); (**f**) Harmonic and percussive components \(x_{\mathrm{h}}\) and \(x_{\mathrm{P}}\).