# Open-Set Recognition of Pansori Rhythm Patterns Based on Audio Segmentation
## 3 Methodology
### The Lightweight Encoder-Decoder Segmentation Network  
Our methodology is a lightweight encoder-decoder segmentation architecture, where the encoder is tasked with extracting features from 2-D log-Mel spectrograms, and the decoder segment makes 1-D temporal decisions following feature mapping in the mapping block.  
Figure 1: The pipeline of open-set recognition for Pansori rhythm patterns.  
An audio waveform graphically represents the variations in sound pressure along the temporal axis. The Short-Time Fourier Transform (STFT) Ref. [39] is applied to the audio signal after temporal sampling and amplitude quantization. This transformation expresses each signal frame as spectral data, displaying the relationship between frequency and energy. This frequency-energy relationship is visualized in a spectrogram. To accommodate the nonlinear frequency sensitivity of the human ear, this spectrogram is transformed into a Mel-frequency scale using \(mel(f)=-2595\times log_{10}(1+f/700)\). Subsequently, the magnitude of each Mel-frequency bin is converted to a logarithmic scale. The resulting log-Mel spectrogram plots time against Mel-frequency, akin to the spectrogram, and can be interpreted as a 2-D image with dimensions \(W\times T\), in which W represents the width in terms of Mel-frequency bins, and the height is \(T=D\times Sr/hop\_length\), where D stands for the physical analysis duration in seconds, \(Sr\) is the sampling rate, and hop_length means the hop length of the successive windows for STFT.  
Figure 2 illustrates the structure of the proposed lightweight encoder-decoder network. The encoder functions as a 2-D feature extractor, employing multiple convolutional layers similar to those used in image analysis. The input to this system is a single-channel log-Mel spectrogram patch sized \(W\times T\), rather than a traditional three-color channel image. During the training phase, the quantity of patches is increased through random cropping along the temporal axis, enhancing the robustness and variability of the training data.  
In the experimental setup, we employed several backbone architectures for the 2-D encoder, including the HRNet series Ref. [40], the ResNet series Ref. [41], and Wide ResNet Ref. [42]. The output of the encoder diminishes the dimensions of the feature maps to \(W/32\times T/32\), while augmenting the number of features to \(N\). B represents the batch size used during training, as shown in Figure 2.  
The decoding process commences with 1-D max pooling along the Mel-frequency axis, followed by a sequence upsampling block that performs temporal interpolation to restore the temporal resolution to that of the input, using nearest-neighbor interpolation along the T-axis. Subsequently, the number of features is reduced from \(N\) to \(\hat{N}\) in the PostBlock, as depicted in Figure 2. The PostBlock consists of \(\hat{N}=64\) filter, incorporating two blocks of consecutive 3 \(\times\) 1 convolutions, batch normalization, and PReLU nonlinear activation functions. The streamlined feature set is then utilized for class determination via a \(1\times 1\) convolution layer, followed by a softmax operation. It is noteworthy that the temporal resolution of the decisions corresponds to the STFT hop size; however, due to the deep layer structure, the encoder encompasses a considerably larger number of frames, increasing the receptive field for each decision. Moreover, the architecture can be conceptualized as a fully convolutional network, akin to those used for semantic image segmentation. Unlike the traditional 2-D pixel-by-pixel output, this structure generates a frame-wise 1-D temporal  
Figure 2: The structure of the open-set audio-segmentation network.  
segmentation output from the 2-D log-Mel spectrogram dimensions, \(W\times T\), illustrating its adaptability to both audio and visual data formats.