# Multi-View Ground-Based Cloud Recognition by Transferring Deep Visual Information
## 3 Approach
### Weighted Metric Learning  
Suppose there is a sample pair \((i,z)\), where \(i\in\mathbb{R}^{d\times 1}\) and \(z\in\mathbb{R}^{d\times 1}\) are the feature vectors of two cloud images from two views, respectively (i.e., \(i\) and \(z\) come from two feature spaces). If the category labels of \(i\) and \(z\) are the same (or different), we define \((i,z)\) as a similar pair (dissimilar pair). The number of cloud categories from each view is \(N\), and we further construct \(N\) sets of similar pairs:  
\[C_{n}:(i,z)\in C_{n},\ (n=1,2,...,N) \tag{2}\]  
Figure 4: Each PSM is divided into \(2\times 2\) regions, which are denoted as four colors, i.e., blue, green, yellow, and pink, respectively. We extract features from each region, and apply max pooling for the final feature representation. where \(C_{n}\) is a set of similar pairs in the \(n\)-th category. We formulate the dissimilar pairs as:  
\[I:(i,z)\in I \tag{3}\]  
We aspire to learn a transformation matrix \(M\in\mathbb{R}^{d\times r}\) (\(r\leq d\)) to parameterize the squared Mahalanobis distance:  
\[D_{M}(i,z)=(i-z)^{T}M(i-z) \tag{4}\]  
where \(M=GG^{T}\) is a positive semidefinite matrix. For convenience, we denote \(s=(i-z)\). The squared Mahalanobis distance is a scalar, and hence we reformulate Equation (4) as:  
\[ \begin{split} D_{M}(i,z)&=s^{T}Ms\\ &=\mathit{Tr}(s^{T}GG^{T}s)\\ &=\mathit{Tr}(G^{T}ss^{T}G)\end{split}  \tag{5}\]  
Our goal is to minimize the distance between similar pairs, and meanwhile maximize the distance between dissimilar pairs. For this purpose, we conduct the following objective function:  
\[ \begin{split}&\min_{M}\;D_{C}-D_{I}\\ & s.t.\;M\geq 0\\ &\mathit{Tr}(M)=1\end{split}  \tag{6}\]  
where \(D_{C}-D_{I}\) is the cost function, the distances of all similar pairs are added to obtain \(D_{C}\), and \(D_{I}\) is the sum of the distances of dissimilar pairs. \(D_{C}\) and \(D_{I}\) are defined in the following. The first constraint ensures a valid metric, and the second one excludes the trivial solution [42]. When computing \(D_{C}\) in the learning process, the classical metric learning methods assign the same weight to each similar pair of all categories. This does not consider that the numbers of similar pairs in each category is largely unbalanced. This weight strategy is not suitable for multi-view ground-based cloud recognition, because the occurrence probabilities of various weather conditions are different, and the number of cloud images in each category varies greatly resulting in the unbalanced similar pairs. Therefore, we propose WML to solve the problem of sample unbalance. For similar pairs, we assign a different weight to each category. Concretely, we first compute the distances between similar pairs of each category, and give a weight to each category according to the similar pair number. Then we sum the weighted distance of all categories. We compute \(D_{C}\) and \(D_{I}\) by:  
\[D_{C}=\sum_{n=1}^{N}\frac{1}{|C_{n}|}\sum_{(i,z)\in C_{n}}\mathit{Tr}(G^{T}ss^{ T}G) \tag{7}\]  
\[D_{I}=\frac{1}{|I|}\sum_{(i,z)\in I}\mathit{Tr}(G^{T}ss^{T}G) \tag{8}\]  
where \(|C_{n}|\) is the number of similar pairs in the \(n\)-th category, and \(|I|\) is the total number of dissimilar pairs of all categories. We minimize the objective function, i.e., Equation (6), subject to two constraints to learn \(M\). Since \(M=GG^{T}\) is a positive semidefinite matrix, the first constraint can be relaxed when explicitly solved for \(M\)[42].