# Audio Time Stretching Using Fuzzy Classification of Spectral Bins
## 2 Fuzzy Classification of Bins in the Spectrogram  
The proposed method for the classification of spectral bins is based on the observation that, in a time-frequency representation of a signal, stationary tonal components appear as ridges in the time direction, whereas transient components appear as ridges in the frequency direction [29; 33]. Thus, if a spectral bin contributes to the forming of a time-direction ridge, most of its energy is likely to come from a tonal component in the input signal. Similarly, if a spectral bin contributes to the forming of a frequency-direction ridge, most of its energy is probably from a transient component. As a time-frequency representation, the short-time Fourier transform (STFT) is used:  
\[X[m,k]=\sum_{n=-N/2}^{N/2}x[n+mH_{a}]w[n]e^{-j\omega_{k}n}, \tag{1}\]  
where \(m\) and \(k\) are the integer time frame and spectral bin indices, respectively, \(x[n]\) is the input signal, \(H_{a}\) is the analysis hop size, \(w[n]\) is the analysis window, \(N\) is the analysis frame length and the number of frequency bins in each frame, and \(\omega_{k}=2\pi k/N\) is the normalized center frequency of the \(k\)th STFT bin. Figure 1 shows the STFT magnitude of a signal consisting of a melody played on the piano, accompanied by soft percussion and a double bass. The time-direction ridges introduced by the harmonic instruments and the frequency-direction ridges introduced by the percussion are apparent on the spectrogram. The tonal and transient STFTs \(X_{s}[m,k]\) and \(X_{t}[m,k]\), respectively, are computed using the median filtering technique proposed by [PERSON] [29]:  
\[X_{s}[m,k]=\text{median}(|X[m-\frac{L_{t}}{2}+1,k]|,...,|X[m+\frac{L_{t}}{2},k]|) \tag{2}\]  
and  
\[X_{t}[m,k]=\text{median}(|X[m,k-\frac{L_{f}}{2}+1]|,...,|X[m,k+\frac{L_{f}}{2} ]|), \tag{3}\]  
where \(L_{t}\) and \(L_{f}\) are the lengths of the median filters in time and frequency directions, respectively. For the tonal STFT, the subscript \(s\) (denoting sinusoidal) is used and for the transient STFT the subscript \(t\). Median filtering in the time direction suppresses the effect of transients in the STFT magnitude, while preserving most of the energy of the tonal components. Conversely, median filtering in the frequency direction suppresses the effect of tonal components, while preserving most of the transient energy [29]. The two median-filtered STFTs are used to estimate the tonalness, noisiness, and transientness of each analysis STFT bin. We estimate tonalness by the ratio  
\[R_{s}[m,k]=\frac{X_{s}[m,k]}{X_{s}[m,k]+X_{t}[m,k]}. \tag{4}\]  
Figure 1: Spectrogram of a signal consisting of piano, percussion, and double bass. We define transientness as the complement of tonalness:  
\[R_{t}[m,k]=1-R_{s}[m,k]=\frac{X_{t}[m,k]}{X_{s}[m,k]+X_{t}[m,k]}. \tag{5}\]  
Signal components which are neither tonal nor transient can be assumed to be noiselike. Experiments on noise signal analysis using the above median filtering method show that the tonalness value is often approximately \(R_{s}=0.5\). This is demonstrated in Figure 2b in which a histogram of the tonalness values of STFT bins of a pink noise signal (Figure 2a) is shown. It can be seen that the tonalness values are approximately normally distributed around the value 0.5. Thus, we estimate noisiness by  
\[R_{n}[m,k]=1-|R_{s}[m,k]-R_{t}[m,k]|= \begin{cases}2R_{s}[m,k],&\text{if }R_{s}[m,k] \leq 0.5\\ 2(1-R_{s}[m,k]),&\text{otherwise}.\end{cases}  \tag{6}\]  
The tonalness, noisiness, and transientness can be used to denote the degree of membership of each STFT bin to the corresponding class in a fuzzy manner. The relations between the classes are visualized in Figure 3.  
Figure 4 shows the computed tonalness, noisiness, and transientness values for the STFT bins of the example audio signal used above. The tonalness values in Figure 4a are close to 1 for the bins which represent the harmonics of the piano and double bass tones, whereas the tonalness values are close to 0 for the bins which represent percussive sounds.