# Exploring Channel Properties to Improve Singing Voice Detection with Convolutional Neural Networks
## 2 Methods
### Channel Attention
#### 2.1.2 Related Work  
We organize the related work in terms of three aspects: the origin of attention mechanism, the attention mechanism in the RNNs, and the attention mechanism in the CNNs. 1. The origin of attention mechanism To the best of our knowledge, the first work in which the attention mechanism was associated with the Neural Network (NN) was undertaken by [PERSON]. In [23], he proposed the Neural Turing Machine (NTM), in which a content-based addressing mechanism was employed to impose \"blurry\" read and write operations on the \"working memory\". Taking the read operation as an example, the operation at the time \(t\) can be simply formulated as the following equation: \[R_{t}=softmax(g(k,M))M\] (1) where \(M=(m_{1},\,m_{2},\,\dots\,\,m_{n})\) is the row vector of the memory contents, \(k\) is the key vector produced by the read head, and \(g\) is a cosine function to measure the similarity between the key vector \(u\) and the content \(v\): \[g(u,v)=\frac{u\cdot v}{||u||\cdot||v||}\] (2)  
Other than the traditional read operation, in which the address and the returned value are fixed at time \(t\), the address is blurry, depending on the distribution on all the memories, and the returned value is contingent on all the memory contents and their corresponding weights. This addressing mechanism essentially is an attentional process in which the weight refers to the degree of attention and the cosine similarity is used to measure the degree. Many attention mechanisms have emerged since [PERSON] NTM was proposed, especially in the Natural Language Processing (NLP) field with RNNs. 2. The attention mechanism in the RNNs In the NLP field, [PERSON] et al. [24] proposed an additive attention model to score how well the input words around position \(j\) and the output at the position \(i\) matched, and then the annotations were computed by these scores. Finally, the context vector for each target word was obtained by the weighed sum of these annotations. [PERSON] et al. [25] further presented a series of attention mechanisms to address the problem, categorized as the global attention and the local attention. The former placed the attention on all the source words, as in the approach of [PERSON] et al., but with some different perspectives, whereas the latter solely focused on the small window of inputs which were differentiable. [PERSON] et al. [26] proposed the other well-known attention mechanism, named self-attention. As the word \"self\" implies, the attention created a mutual relationship among the source words, which was different from the relationship of the other mechanisms, and  
Figure 3: Channel attention mechanism. was described as being between the target words and the source words. In this mechanism, the scaled dot-product (SDP) was utilized to measure the degree of the relationship:  
\[\text{Attention}(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{3}\]  
where \(Q\), \(K\), and \(V\) represent the matrix form of query, key, and value, respectively, and \(d_{k}\) is the key dimension. Similar to the content-based addressing mechanism [23], SDP first calculates the similarity measure and then transforms this measure to the weight matrix by the _softmax_ function. Finally, \(V\) is weighted by the attention. Moreover, to learn more relative information, they offered a multi-head model to represent the different subspaces:  
\[ \begin{split}\text{Multihead}(Q,K,V)&=\text{ Concat}(\text{head}_{1},\cdots,\text{head}_{h})W^{O}\\ \text{where head}_{i}&=\text{Attention}(QW^{Q}_{i },KW^{K}_{i},VW^{V}_{i})\end{split}  \tag{4}\]  
3. The attention mechanism in the CNNs  
To date, most of the attention mechanisms have been designed for RNNs. As another an important variant of DNNs, the CNN plays a signficant role in audio and video processing. Can we place attention on the CNN and improve its performance? [PERSON] et al. [27] proposed a squeeze-and-excitation block to recalibrate the relations among the channels, and achieved great success in a recent image classification task.