# A Deep Curriculum Learning Semi-Supervised Framework for Remote Sensing Scene Classification
## 3 Proposed Method
### Deep Feature Fusion  
Feature fusion refers to the integration and fusion of feature information obtained from different data sources, different feature spaces, or different feature extraction algorithms to improve the performance of tasks such as pattern classification, recognition, and detection [14]. The purpose of CCA is to find two sets of projection directions to maximize the correlation between two random vector projections [13]. In recent years, to overcome the shortcomings of ignoring category information in the CCA method, DCA maximizes the correlation of features of the same sample and minimizes the correlation between features of different types of samples, which is also beneficial in improving the classification performance [14].  
Assume that the number of categories in the dataset is \(c\). Let \(x_{ij}\in X\) denote a deep feature corresponding to the \(j^{th}\) image in the \(i^{th}\) class, which is extracted from one pretrained network. Let \(\overline{x_{i}}\) denote the means of the \(i^{th}\) class: \(\overline{x_{i}}=\frac{1}{n_{i}}\sum_{j=1}^{n}x_{ij}\). Let \(\overline{x}\) denote the means of all classes: \(\overline{x}=\frac{1}{n_{i}}\sum_{i=1}^{n}n_{i}\overline{x_{i}}\), where \(n_{i}\) denotes the number of samples in each class.  
The between-class scatter matrix is defined as  
\[S_{bx(p\times p)}=\sum_{i=1}^{c}n_{i}(\overline{x_{i}}-\overline{x})(\overline {x}-\overline{x_{i}})^{T}=\Phi_{bx}\Phi_{bx}^{T} \tag{1}\]  
where \(\Phi_{bx}=[\sqrt{n_{1}}(\overline{x_{1}}-\overline{x})],[\sqrt{n_{2}}( \overline{x_{2}}-\overline{x})],\dots,[\sqrt{n_{c}}(\overline{x_{c}}-\overline {x})]\).  
To separate the classes, matrix \(\Phi_{bx}^{T}\Phi_{bx}\) should be a diagonal matrix, so the maximum \(r\) eigenvectors of \(S_{bx}\) can be calculated.  
\[(\Phi_{bx}q)^{T}S_{bx}(\Phi_{bx}q)=\Lambda_{r\times r} \tag{2}\]  
where \(\Lambda\) is the diagonal matrix of real non-negative eigenvalues and \(r\) is the largest nonzero eigenvector matrix that is represented by \(\varphi_{(c\times r)}\).  
First, find a mapping matrix \(W_{bx}\) for \(S_{bx}\)[14], so that the between-class scatter matrices are unitized:  
\[W_{bx}^{T}S_{bx}W_{bx}=I \tag{3}\]  
\[X_{(r\times n)}^{\prime}=W_{bx(r\times p)}^{T}X_{(p\times n)} \tag{4}\]  
where \(X^{\prime}\) is the projection of \(X\), and \(S_{bx}\) is the between-class scatter matrix.  
Furthermore, for the second deep feature, \(Y\), which is extracted from another pre-trained network, carry out the following:  
\[W_{by}^{T}S_{by}W_{by}=I \tag{5}\]\[Y^{\prime}_{(r\times n)}=W^{T}_{by(r\times p)}Y_{(r\times n)} \tag{6}\]  
where \(r\) is the rank of the transformed feature [14], \(r\leq min(c-1,rank(X),rank(Y))\).  
Then, to maximize the correlation between features across the two feature sets, the covariance matrix \((S^{\prime}_{xy}=X^{\prime}Y^{\prime T})\), formed by \(X^{\prime}\) and \(Y^{\prime}\), is diagonalized. To solve this problem, we can use the method of singular value decomposition.  
\[S^{\prime}_{xy(r\times r)}=U\Sigma V^{T}\Rightarrow U^{T}S^{\prime}_{xy}V= \Sigma\Rightarrow(U\Sigma^{-\frac{1}{2}})^{T}S^{\prime}_{xy}(V\Sigma^{-\frac{ 1}{2}})=I \tag{7}\]  
Let \(W_{cx}=U\Sigma^{-\frac{1}{2}}\) and \(W_{cy}=V\Sigma^{-\frac{1}{2}}\); then, we can find  
\[DX=W^{T}_{cx}X^{\prime}=W^{T}_{cx}W^{T}_{bx}X=W^{T}_{x}X \tag{8}\]  
\[DY=W^{T}_{cy}Y^{\prime}=W^{T}_{cy}W^{T}_{by}Y=W^{T}_{y}Y \tag{9}\]  
where \(W_{x}=W_{bx}W_{cx}\) and \(W_{y}=W_{by}W_{cy}\) are the final transformation matrices for \(X\) and \(Y\), respectively.