# Empirical Evaluation on Utilizing CNN-Features for Seismic Patch Classification
## 3 Methods and Data
### Classification Methods  
The classification system to be developed in this paper (named HYB) is a method of **hybridizing** an existing linear (or nonlinear) classifier (e.g., an SVM) with a CNN. HYB is a classification framework that performs feature extraction on CNNs, and then trains SVMs using the extracted features. In this approach, when learning the CNN in the target domain, the problem of insufficient learning data can be avoided through transfer learning, using a model learned in advance in the source domain. The CNN architecture extracting the feature vector consists of an input layer, a hidden layer, and an FC layer. The hidden layer includes layers that perform convolution and subsampling. In this structure, the feature vectors we are trying to extract are made from input neurons directly connected to the FC layer. The extracted vector is a midlevel representation, just before the input image passes through the hidden layers and is transferred to the FC layer. Therefore, the dimension of the feature vector is equal to the number of neurons that make up the first FC layer. As a result, we can adjust the number of FC input neurons to find the optimal dimension for the given application. To extract CNN-features as described above, the weights of the CNN that make up the C layers should be fixed in advance. To this end, transfer learning can be used [16]. In transfer learning, the cardinality of the target task's dataset is usually less than that of the source task's training dataset. Because of this, the proposed CNN-feature extraction method can avoid the general difficulty of CNN learning, which requires a large amount of data. Moreover, CNN training in the transfer learning mode is made up of fine-tuning. Therefore, the learning time can be significantly reduced with a small amount of fine-tuning. The CNN model is learned by repeatedly performing forward propagation (FP) and backward propagation (BP), in an end-to-end manner. At this time, one of sum-, max-, or O2P-pooling is used to reduce the amount of processed data. When using O2P-pooling, the learning process is performed as follows: First, for the FP, it is assumed that a feature tensor, \(\mathbf{X}\in\mathbb{R}^{h\times w\times d}\), of the height, \(h\), width, \(w\), and channel, \(d\), is generated from the last C layer, and that this tensor is reshaped into a matrix, \(\mathbf{X}\in\mathbb{R}^{n\times d}\), consisting of \(n=hw\) features of the \(d\)-dimension. Then the second-order pooling, \(\mathbf{\Sigma}\in\mathbb{R}^{d\times d}\), for \(\mathbf{X}\) is computed as:  
\[\mathbf{\Sigma}=\mathbf{X}\mathbf{1}\mathbf{X}^{\mathbf{T}}, \tag{4}\]  
where \(\mathbf{\tilde{\mathbf{\Gamma}}}=\frac{1}{n}\left(\mathbf{1}-\frac{1}{n}\mathbf{1}\right)\); \(\mathbf{1}\) and \(\mathbf{1}\) are the \(n\times n\) identity matrix and the matrix of all the ones, respectively. To improve the representation power, \(\mathbf{\Sigma}\) is transformed into \(\mathbf{Z}\) by performing matrix power normalization using SVD (singular value decomposition) and is then sent to the FC layer. Next, BP proceeds in the opposite direction. Given the gradient of a loss function, \(l\) w.r.t. \(\mathbf{Z}\), propagated from the FC layer, i.e., \(\partial l/\partial\mathbf{Z}\), the gradient of \(l\) w.r.t. \(\mathbf{\Sigma}\) is estimated sequentially. First, we have prepared all the relevant derivations (see Equation (2) in [31]),and then we can calculate the \(\partial l/\partial\mathbf{\Sigma}\). Using this derivative, the gradient of \(l\) w.r.t. \(\mathbf{X}\), which is used to update the weights of the network, is determined as:  
\[\frac{\partial l}{\partial X}=\overline{\mathbf{\mathrm{I}}}\mathbf{\mathrm{X}}\Bigg{(} \frac{\partial l}{\partial\mathbf{\mathrm{\Sigma}}}+\bigg{(}\frac{\partial l}{ \partial\mathbf{\mathrm{\Sigma}}}\bigg{)}^{T}\Bigg{)}. \tag{5}\]  
Here, it should be noted that the BP contains GPU-nonfriendly computations, such as SVD, which leads to costly training.