# Korean Passori Vocal Note Transcription Using Attention-Based Segmentation and Viterbi Decoding
## 3 Method
### Frame-Level Transcription
#### 3.1.2 Attention-Based Semantic Segmentation Framework  
To conduct frame-level transcription for vocal melody extraction in the source task, we use encoder-decoder architecture which was originally developed for image segmentation [32]. The encoder takes an input CFP that represents both the frequency and periodicity information of the signal. The input has three channels that can represent the three aspects of music features. First, the input dimension of size \(3\times F\times T\) is passed into the encoder consisting of three convolution blocks. Each of the convolution blocks consists of a batch normalization layer, 2D convolution layer, scaled exponential linear unit (SELU) activation, and max pooling layer. The pooling layer has a size of \(4\times 1\) which pools only in the frequency dimension and consists of pooling indices between the pooling and unpooling layers. The final feature of the encoder is passed into the three consecutive self-attention blocks. Each self-attention block consists of a linear transformation of query, key, and value tensor applied to the output of the encoder. The attention weights are computed using the scaled dot-product attention mechanism between query and key. The softmax function is applied to obtain the normalized attention weights. The attention weights are multiplied with the value tensor to obtain the final attended values from the self-attention block. The output of the self-attention-based features has two paths, one for the decoder and another for the convolutional block. The path for the decoder consists of three blocks of convolution which are the same as the encoder and is used to upsample the features using transposed convolution. This path is constructed to detect the voicing frames of the audio. Similarly, another path for a single convolutional block is designed to estimate the non-voicing frames of the audio. As we apply the pooling operation only in frequency dimension, the size of this non-voicing detector is \(1\times T\). Finally, we concatenate the detection obtained from the two different paths to obtain the final output representation of vocal melody extraction. The architecture for the encoder-decoder network and self-attention-based vocal melody extraction is shown in Figures 1 and 2, respectively. The final output of the model architecture is obtained by concatenating the result of the bottom block and the upsampled result from the last convolutional block of the decoder along the frequency dimension (\(dim=2\)). The softmax function is then applied along this frequency dimension to obtain the final output, as shown in Equation (1). \[output=softmax(torch.cat(bottom(pool3(conv3(pool2(conv2(pool1(conv1(x))))))), dim=2)) \tag{1}\]  
We also integrate the channel attention mechanism after obtaining the final features of the encoder network and compare the performance with self-attention. The channel attention module introduces a channel-wise attention mechanism by selectively amplifying informative channels within the input tensor. This mechanism is particularly beneficial for tasks such as vocal melody extraction, where certain channels may carry more crucial pitch information. To apply the channel attention, first, an adaptive average pooling operation is applied to reduce the spatial dimensions of the tensor to \(1\times 1\), which can effectively summarize the global information across the channel dimension. Subsequently, the tensor passes through two linear fully connected layers. The first linear layer reduces the dimensionality of the channels to \(input\_channels/reduction\ ratio\), followed by ReLU activation, where we keep the reduction ratio as 16. The tensor is then further processed by another linear layer, which restores the dimensionality of the original number of channels. The final attention weights are reshaped, which allows for element-wise multiplication with the original input obtained from the feature map of the encoder.