# Fusion Classification Method Based on Audiovisual Information Processing
## 3 Proposed Method
### The Conversion of the Spectrogram (Audio Preprocessing)  
Considering that the acquisition process of sound will be affected by some environmental factors, and the acquisition of sound characteristics will be affected by large noise, we use multiple microphone arrays in terms of hardware and can use beamforming technology to lock the direction of sound, which has a good signal-to-noise ratio.  
The sound captured by the microphone array is a one-dimensional time series signal. First, we will send the sound signal via a high-pass filter, specifically Equation (1):  
\[y(t)=x(t)-ax(t-1). \tag{1}\]  
Among them, the filter coefficient (\(a\)) is set to 0.95, \(t\) represents the time of the sound signal, \(x(t)\) represents the input signal, and \(y(t)\) represents the output signal. In this manner, we complete the pre-emphasis processing of the input sound signal and compensate for the high-frequency part of the speech signal that was suppressed by the pronunciation signal system.  
Nevertheless, the sound signal exhibits stationary characteristics over a brief period of time, allowing us to simply apply a framing operation to the output sound signal produced by the high-pass filter. To enhance the signal's smoothness, we multiply each frame of the sound signal by a hamming window, thereby increasing the continuity from the left end of the frame to the right end. Suppose that the signal after framing is \(S(n)\), \(n=0\), 1, 2,..., \(\mathrm{N}-1\), where \(\mathrm{N}\) is the length of the window. Hamming window makes  
Figure 1: M-AVFA general frame diagram.  
the edge of the signal gradually weaken by weighting the signal, reducing the leakage of signal energy into the adjacent frequency components, that is, reducing the aliasing between different frequency components, thus suppressing the influence of noise. The signal of each frame is multiplied by the Hamming window, denoted as Equation (2):  
\[S^{\prime}(n)=S(n)\times W(n). \tag{2}\]  
Among them, \(S(n)\) represents a sub-frame defined by \(y(t)\) and \(W(n)\) is a Hamming window, whose form is defined by Equation (3):  
\[W(n)=0.54-0.46\cos\frac{2\pi n}{L},\ 0\leq n\leq L-1, \tag{3}\]  
where \(W(n)\) represents the weight value of the signal at the nth discrete time point of the window, \(L\) represents the length of the window, and \(n\) represents the sample index of the window.  
The energy distribution in the frequency domain observes its characteristics, and distinct speech characteristics can be distinct through different energy distributions in the frequency domain. We perform an \(N\)-point FFT on each frame of the frame windowed signal to calculate the spectrum. Equation (4) is as follows:  
\[S_{i}[k]=\sum_{m=1}^{N}S^{\prime}(n)W(n)e^{-j2\pi}\frac{n}{N},\ 1\leq n\leq N, \tag{4}\]  
where \(S^{\prime}(n)\) represents the input frame data, \(W(n)\) refers to the Hamming window function, \(N\) is the number of Fourier transform points, and \(S_{i}[k]\) denotes the frequency domain expression of the input audio signal. The absolute values of the Fourier transform results are used to obtain the spectral amplitude.  
The spectral amplitude of each time frame is arranged into a two-dimensional matrix according to Equation (5):  
\[I_{audio}[i,k]=|S_{i}[k]|, \tag{5}\]  
where \(i\) represents the index of the time frame, \(k\) represents the index of the frequency, and \(I_{audio}[i,k]\) is a two-dimensional matrix. This is the spectrogram of sound that we need.  
Therefore, this paper completes the conversion of one-dimensional acoustic signal to two-dimensional spectrogram signal. Figure 2 shows the spectrum diagram.