# A Deep Curriculum Learning Semi-Supervised Framework for Remote Sensing Scene Classification
## 3 Proposed Method
### Multimodal Curriculum Learning
In order to evaluate the effect of the \(t\)-th learning, the learning confidence is evaluated by the entropy of \(W(t)\) label matrix \(F_{W(t)}\), which is formally defined by  
\[G(F_{W(t)})=\cos(\frac{\pi}{2}\cdot H(F_{W(t)})/W(t)) \tag{16}\]  
\[H(F_{W(t)})=\Sigma_{i}\Sigma_{j}(F_{W(t)})_{ij}\log_{c}(F_{W(t)})_{ij} \tag{17}\]  
where \(G(F_{W(t)})\) is the learning evaluation function, \(H(F_{W(t)})\) is entropy, and \(G(F_{W(t)})\) is in the range [0, 1]. The number of samples in the \((t+1)\) th curriculum is defined as:  
\[W^{(t+1)}=\min(maxCnum,b^{(t+1)}\cdot G(F_{W(t)})) \tag{18}\]  
where \(maxCnum\) is the maximum number of learning curricula, and \(b^{(t+1)}\) is the size of the set in the \((t+1)\) th iteration. The number of curricula to be learned next time is determined by two factors: the effect of the students' learning in this iteration and the maximum acceptable number of curricula. This is because, even in an easy curriculum, if there are a large number of curricula, it will lead to a decrease in learning efficiency. Therefore, we define the upper limit of the number of curricula for the next iterative learning, which is defined as \(maxCnum=100\) in the experiment. The idea of a multimodal curriculum learning method is this: if each modality is considered a teacher, and multimodality is considered as a plurality of different teachers, each teacher independently recommends the curriculum he considers, and then integrates each teachers' suggestions for the recommend curricula for students to learn, as shown in Figure 2.  
In this paper, each modal recommended curricula are intersected to obtain the recommended curricula, and the learning feedback is used to determine the number of curricula (W in Figure 2) to be learned in the next iteration. \[W=W^{(m_{1})}\cap W^{(m_{2})}\ldots\cap W^{(m_{n})} \tag{19}\]  
where \(W^{(m_{n})}\) denote the set of curricula in the n-th modality. The DCLSSF model is described in Algorithm 1. This method is suitable for datasets with a small data size, because each sample is a node of the graph, and the number of edges in the graph will increase geometrically as the data size increases. In addition, if the label contains a noisy label case, the performance of this method is also poor, because labeled images with potentially mislabeled images are difficult and should be reclassified in later propagation. ```
0: Remote sensing images \(I=[I_{1},\ldots,I_{n}]\); Pretrained CNN models \(P=[P_{1},\ldots,P_{p}]\);
0: Accuracy;
1: // Deep features extraction
2: for all \(1\leq i\leq n\) do
3: Extract deep feature \(X_{i}^{v}\) from \(I_{i}\) using pretrained CNN model \(P_{o}\), \(X_{i}^{v}=[x_{1}^{v},\ldots,x_{n}^{v}]\);
4: end
5: // Deep features fusion
6: Deep features are fused in pairs to form features of different models by (7) and (8);
7: // Multimodal curriculum learning
8: Each modal feature is fed into the multimodal curriculum learning network;
9: For unlabeled samples, select easy courses for label propagation by (11);
10: Based on the learning confidence of each learning by (13), learning feedback is given to the next curriculum learning by (15);
11:return Accuracy;

Figure 2: Multimodal curriculum learning. The labeled images and unlabeled images are represented by green and yellow balls, respectively. Among the unlabeled images, the recommended curriculum is surrounded by red ring. ## 4 Experimental Results

To verify the effectiveness of DCLSSF, it was evaluated on three public remote sensing image datasets. The following is an introduction to the datasets and a discussion of the results of each dataset. The three popular benchmark datasets used are the UC Merced dataset [25], the AID dataset [26], and the NWPU RESICS-45 dataset [27]. The UC Merced Land Use dataset (referred to as UC Merced) includes 2100 aerial optical images. In recent years, many researchers have used this dataset, allowing extensive comparisons of results with the literature.