# A Topical Category-Aware Neural Text Summarizer
## 3 Class Activation Map for Text-CNN  
### Text-CNN  
Convolutional neural networks (CNN) have been mainly applied to image processing. Affected by great success of CNN in many applications of computer vision, [PERSON] recently proposed the text-CNN [25] to apply CNNs to text classification such as sentiment analysis and sentence classification. The text-CNN consists of a single convolution layer, a single max pooling layer, and one or more fully connected layers.  
Let \(\textbf{X}=\langle x_{1},x_{2},\ldots,x_{n}\rangle\) be a text where \(x_{i}\) is the \(i\)-th word of **X**, and \(\textbf{x}_{i}\in\mathbb{R}^{d}\) be the \(d\)-dimensional vector representation to \(x_{i}\). When \(\textbf{x}_{i:i+k_{h}-1}\) is a word window of size \(k_{h}\) from \(x_{i}\) to \(x_{i+k_{h}-1}\), it is represented as  
\[\textbf{x}_{i:i+k_{h}-1}=\textbf{x}_{i}\oplus\ldots\oplus\textbf{x}_{i+k_{h}-1},\]  
where \(\oplus\) is the concatenation operator. The convolution layer creates a feature map **C** of which element is again a vector generated by sliding **X** with a \(k_{h}\)-size window and a kernel, where there are \(H\) kinds of window sizes and \(J\) kinds of kernels. That is, \(\textbf{c}_{j}^{k_{h}}\), the element of **C** with a window size \(k_{h}\) (\(1\leq h\leq H\)) and the \(j\)-th (\(1\leq j\leq J\)) kernel is  
\[\textbf{c}_{j}^{k_{h}}=\left[\begin{matrix}\begin{matrix}  
Figure 1 depicts the general architecture of CAM. This architecture is identical to the text-CNN except the global average pooling layer. That is, from the vector **C** in Equation (2), the final feature vector **g** of CAM  
\[\textbf{g}=\left[g_{1}^{k_{1}},\ldots,g_{J}^{k_{1}},g_{1}^{k_{2}},\ldots,g_{J}^{ k_{2}},\ldots,g_{1}^{k_{H}},\ldots,g_{J}^{k_{H}}\right]\]  
is obtained by computing \(g_{j}^{k_{h}}\) as  
\[g_{j}^{k_{h}}=\sum_{u=1}^{n-k_{h}+1}c_{j}^{k_{h}}(u). \tag{3}\]  
The possibility of each class \(\tau\) for the input **X** by CAM is determined through a fully connected layer. That is, the possibility \(S_{\tau}\) is  
\[S_{\tau}=\sum_{h,j}W_{\tau}^{k_{h},j}g_{j}^{k_{h}},\]  
where \(W_{\tau}^{k_{h},j}\) is a weight of \(g_{j}^{k_{h}}\) to a class \(\tau\). Since a bias term does not affect classification performance significantly, there is no bias term in \(S_{\tau}\). CAM is trained in the same manner to training text-CNN with the training set \(D\).  
From Equation (3), \(S_{\tau}\) can be formulated with \(c_{j}^{k_{h}}\)'s as  
\[S_{\tau} = \sum_{h,j}W_{\tau}^{k_{h},j}g_{j}^{k_{h}}\] \[= \sum_{h,j}W_{\tau}^{k_{h},j}\sum_{u}c_{j}^{k_{h}}(u)\] \[= \sum_{u}\sum_{h,j}W_{\tau}^{k_{h},j}c_{j}^{k_{h}}(u).\]  
Figure 1: The architecture of class activation map (CAM) applied to text classification.  
By defining  
\[M_{\tau}(u)=\sum_{h,j}W_{\tau}^{h_{h},j}c_{j}^{h_{h}}(u), \tag{4}\]  
\(S_{\tau}\) becomes \(S_{\tau}=\sum_{u}M_{\tau}(u)\). Thus, \(M_{\tau}(u)\) indicates the importance of a word \(u\) in classifying \(\mathbf{X}\) to class \(\tau\). To complete \(M_{\tau}(u)\), \(\boldsymbol{\zeta}_{j}^{h}\) in Equation (1) should be of the same size regardless of window size. For this, the class activation map is upsampled to the length of the input text \(\mathbf{X}\).  
CNN is known to have an object detection capability in computer vision [26], but is poor at detecting the edge of objects due to its max pooling. On the other hand, the global average pooling enforces CAM to detect the inner side of objects. If CAM is applied to a text, the words or phrases that have a high impact on a target class are found out. Therefore, CAM can be regarded as word weights for the target class.