# A Deep Curriculum Learning Semi-Supervised Framework for Remote Sensing Scene Classification
## 3 Proposed Method
### Multimodal Curriculum Learning  
The curriculum learning aims to choose the easiest curriculum to learn from the curricula. In the teaching process, some of the simplest samples from the unlabeled samples are selected for label propagation. This simple sample can be considered as an easy curriculum. The label propagation process is the teaching process for teachers. After the students have learned the easy curriculum, they will give the teacher feedback on the learning situation so that the teacher can plan the next curriculum. Students' feedback to teachers is expressed by learning confidence. Curriculum selection aims to select the easier curricula, and in order to measure the difficulty of the curriculum, they are defined by reliability. Suppose that there is a sample set \(D=\{x_{1},\cdots,x_{l},x_{l+1},\cdots,x_{n}\}\) composed of \(n=l+p\) samples, where the first \(l\) elements constitute the labeled set \(L\), and the remaining \(p\) samples constitute the unlabeled set \(U\). The curriculum is defined as a set \(W\), which represents \(w\) simple samples selected from a set of unlabeled samples during a propagation process. First, each image \(x_{i}\) is given a random variable \(y_{i}\), and the propagation on the graph is considered a Gaussian process, so this Gaussian process is composed of the random variable \(Y=(y_{1},\ldots,y_{n})^{T}\). \[p(y)\propto\exp(-1/2Y^{T}(L+I/\kappa^{2})Y) \tag{10}\]  
where \(L+I/\kappa^{2}\) is a regularized Laplace matrix, and \(\kappa^{2}=100\). Therefore, the Gaussian process has a compact expression \(Y\sim N(0;\Sigma)\), and its covariance matrix is \(\Sigma=(L+I/\kappa^{2})^{-1}\). If the conditional entropy \(H(y_{W}\mid y_{L})\) is small, then the curriculum is reliable. The \(y_{W}\) and \(y_{L}\) distributions represent the subvectors of \(y\) in the sets \(W\) and \(L\). The smaller the conditional entropy, the easier the curriculum \(W\) is. \[H(y_{W}|y_{L})=\sum_{y_{l}}p(y_{l})H(y_{W}|y_{L}=y_{l}) \tag{11}\]  
_Reliability:_ By leveraging the properties of multivariate Gaussian, the most reliable curriculum can be found by optimizing [7]:  
\[\min_{W\subseteq U}H(y_{W}\mid y_{L}) \tag{12}\]  
In order to find the easiest curriculum, and the most reliable curriculum, optimize the solution of Equation (12). \[ \begin{split} W^{*}&=\min_{W\subseteq U}H(y_{W}\mid y_{L})\\ &\Leftrightarrow\min_{W\subseteq U}H(y_{W\bigcup L})-H(y_{L})\\ \Leftrightarrow\min_{W\subseteq U}(\frac{w+l}{2}(1+\ln 2\pi)+ \frac{1}{2}\ln\mid\Sigma_{w\bigcup L,w\bigcup L}\mid)-(\frac{l}{2}(1+\ln 2 \pi)+\frac{1}{2}\ln\mid\Sigma_{L,L}\mid)\\ \Leftrightarrow\min_{W\subseteq U}(\frac{w}{2}(1+\ln 2\pi)+ \frac{1}{2}\ln\frac{\mid\Sigma_{w\bigcup L,w\bigcup L}\mid}{\mid\Sigma_{L,L} \mid})\end{split}  \tag{13}\]  
where \(\Sigma_{W\bigcup L,W\bigcup L}\) and \(\Sigma_{L,L}\) are sub-matrices of \(\Sigma\), corresponding to their subscripts. \[ \begin{split}\text{Block by matrix }\Sigma_{W\bigcup L,W\bigcup L}= \left(\begin{array}{cc}\Sigma_{W,W}&\Sigma_{W,L}\\ \Sigma_{L,W}&\Sigma_{L,L}\end{array}\right)\text{, then}\\ \Leftrightarrow\min_{W\subset U}\text{tr }\mid\Sigma_{W,W}-\Sigma_{W,L} \Sigma_{L,L}^{-1}\Sigma_{L,W}\mid\end{split}  \tag{14}\]  
Students mainly study sample curricula taught by the teacher, and then, via feedback, send the results to the teacher to decide on the next curriculum to be taught. The adopted iterative model represents the propagation of the labels of the \(t\)-th curriculum \(W^{(t)}\) and the previously learned sample \(W^{(1:t-1)}\). The initial state is in the labeled sample set \(L^{0}\). Under the \(t^{th}\) propagation, the iterative model is:  
\[G_{i}^{(t)}=\left\{ \begin{aligned} & G_{i}^{0}& x_{i}\in L^{0}\\ & Q_{i}G^{(t-1)}& x_{i}\in W^{(1:t-1)}\bigcup W^{(t)} \end{aligned} \right. \tag{15}\]  
where \(Q_{i}\) represents the \(i\)-th row of the transition matrix \(Q\) calculated by \(Q=D^{-1}W\), and \(W^{(1:t-1)}\) represents the set \(W^{(1)}\bigcup\cdots\bigcup W^{(t-1)}\).