# Multimodal Chinese Sarcasm Detection Integrating Audio Attributes and Textual Features
## 3 Methodology
### Feature Extraction Layer
#### 3.2.3 Audio Descriptive Attribute Feature Representation  
There exists a large amount of sarcasm-related information in audio that can be further explored, but current research mainly focuses on clues within the audio itself, without establishing connections between audio and text. Therefore, this paper introduces the concept of audio attributes. Through audio attributes, a bridge is built between audio and textual modalities to enhance synchronization of information across them.  
As shown in Figure 3, we preliminarily extract a large number of sarcasm-related acoustic features using OpenSMILE. These features help the model to understand the relationship between changes in audio signals and the semantic content in text, and they are crucial for capturing subtle emotional variations in audio.  
\[X=[F0,\text{loudness},\text{shimmer},\text{jitter},\text{HNR},F1,F2] \tag{7}\]  
The audio attribute signals selected in this work include two levels of descriptors. On one hand, low-level descriptors such as F0, shimmer, and jitter reflect fine-grained fluctuations in audio signals. On the other hand, high-level features like HNR can be used as more direct indicators to detect sarcastic speech.  
However, due to the different dimensional ranges across modalities, it is necessary to process the extracted features to make them suitable for our model framework. In this study, we scale all attributes to a fixed range to ensure that each dimension contributes properly. After normalization, the audio attribute features become a kind of textual representation that can describe audio signals, thereby improving cross-modal interaction and enhancing the model's ability to analyze multimodal data.  
In this study, we select seven acoustic attributes--F0, loudness, shimmer, jitter, HNR, F1, and F2--as the basis for audio feature extraction. These attributes were chosen because they jointly capture both low-level prosodic variations and high-level vocal characteristics, which are essential for modeling subtle emotional cues in sarcastic speech. F0, shimmer, and jitter primarily describe fine-grained pitch fluctuations, while HNR reflects speech signal periodicity and noisiness. F1 and F2 formant frequencies provide information about  
Figure 3: Feature integration layer.  
-vocal tract resonances and articulation style. Compared to traditional features such as MFCCs, these selected attributes offer a more direct and interpretable representation of emotional and prosodic nuances, which are critical in sarcasm understanding.  
\[\text{normalized\_feature}_{i}=\frac{\text{feature}_{i}-\text{min}(\text{ feature}_{i})}{\text{max}(\text{feature}_{i})-\text{min}(\text{feature}_{i})} \tag{8}\]  
To facilitate downstream multimodal fusion, the output of the BERT encoder is further refined through a batch normalization and activation layer, and subsequently projected to a fixed dimensionality of 128 via a linear transformation. The resulting audio-attribute embedding vector, denoted as \(H_{\text{audio\_desc}}\), is 128-dimensional and serves as the final representation used in cross-modal integration.  
Unlike conventional multimodal fusion strategies that treat text and audio features independently until the final integration stage, our approach embeds audio-attribute vectors into the same semantic space as text. By encoding the attribute features via a BERT encoder, the audio cues are aligned with the token-level semantics of textual inputs. This unified representation enables fine-grained cross-modal interactions, allowing the model to associate prosodic variations in audio with contextual semantics in text. Such design not only enhances fusion interpretability but also highlights the novelty of leveraging attribute-aware alignment in sarcasm detection [34].