## 3 The Proposed EFAFN Model
### Multi-Modal Feature Extraction  
Text Features: we use BERT [12] to process the text data in the dataset into a unique feature vector. First, we remove the data with empty text in the dataset, then input the text into the \(BERT_{base}\) model and average the outputs of the last \(L=4\) transformer layers in the \(BERT_{base}\) model as the feature vector of each text. Finally, each piece of text will be represented by a \(d_{t}=768\) dimensional feature vector \(u_{t}\). \[\left\{u_{i}^{t}\right\}_{i=1}^{L}=BERT_{base}(T) \tag{1}\]  
\[u_{t}=\frac{1}{L}\left(\sum_{i=1}^{L}u_{i}^{t}\right)\in\mathbb{R}^{d_{t}} \tag{2}\]  
where \(u_{i}^{t}\) represents the output of the last i-th transformer layer in the BERT-base model, and \(T\) represents a piece of text. Figure 2: The whole architecture of the network we proposed. It includes three key steps: (**a**) multimodal feature extraction, (**b**) adaptive feature fusion, and (**c**) feature classification. Among them, BERT represents a self-encoding language model presented by [PERSON] et al. [12], SFEN represents the speech feature extraction method presented by [PERSON] et al. [9], and FFEN represents the facial feature extraction network we use, which we will describe in detail in Section 3.2. The adaptive feature fusion strategy is introduced, in fact, in Section 3.3. MLP stands for Multilayer Perceptron and is described in detail in Section 3.4. Speech Features: we adopt the method of extracting features from speech data proposed by [PERSON] et al. [9], and use the speech processing library Librosa, which extracts basic features from audio data. First, we load audio samples into a time series signal with a sampling rate of 22,050 Hz; we use a heuristic sound extraction method to remove background noise. Finally, we divide the audio signal into \(d_{w}\) non-overlapping windows to extract local features, including MFCC, Mel-spectogram, spectral centroid, and their associated temporary derivatives (delta). All the elements are connected to form a joint feature vector \(\left\{u_{i}^{a}\right\}_{i=1}^{d_{w}}\)with \(d_{a}=283\) dimensions for each window. Finally, the joint feature \(u_{a}\) of each piece of audio is obtained by calculating the average value of all windows. \[u_{i}^{a}=u_{i}^{MFCC}\oplus u_{i}^{MFCC\ delta}\oplus u_{i}^{Mel}\oplus u_{i}^{Mel \ delta}\oplus u_{i}^{spec} \tag{3}\]  
\[u_{a}=\frac{1}{d_{w}}(\Sigma_{i}u_{i}^{a})\in\mathbb{R}^{d_{a}} \tag{4}\]  
where \(\oplus\) is the concatenation operator, \(u_{i}^{MFCC}\), \(u_{i}^{MFCC\ delta}\), \(u_{i}^{Mel}\), \(u_{i}^{Mel\ delta}\), and \(u_{i}^{spec}\) represent MFCC features, MFCC associated temporary derivatives, Mel-spectogram features, Mel-spectogram associated temporary derivatives, and spectral centroid of each window, respectively. Face Image Features: as facial information contains emotional information related to sarcasm detection, we believe that the video modality should focus on the person's news rather than the image's background information. The network architecture of facial feature extraction is shown in Figure 3. First, we use the face detection model provided by DLIB ([[http://dlib.net/face_detection_ex.cpp.html](http://dlib.net/face_detection_ex.cpp.html)]([http://dlib.net/face_detection_ex.cpp.html](http://dlib.net/face_detection_ex.cpp.html)), accessed on 13 August 2021) to detect the face \(f_{i}\) of each frame in the video conversation. Let \(D_{i}\) denote the maximum height of the face image seen from each image frame. Since the size of each face detected in the face detection process is inconsistent, the black block \(B_{i}\) is used to fill the vacant part in the splicing process. After the filling operation, the heights of all face images are uniform. Then, a horizontal stitching operation is performed on all faces to obtain the final image input to the network. The formulation is as follows. \[d_{B(f_{i})}=(3,Len(f_{i}),D_{i}-Hei(f_{i})) \tag{5}\]  
\[padding(f_{i})=\left\{