# Deep Unsupervised Embedding for Remote Sensing Image Retrieval Using Textual Cues
## 2 Proposed Methodology
### Optimization  
Retrieval tasks are usually solved by learning a distance metric [43]. Inspired by accomplishments of deep learning in computer vision [26], deep neural networks have been used to learn how to embed discriminative features useful for learning these distances [44; 45]. In this case, the embedded features of similar samples should be closer, while those of dissimilar samples should be farther. The literature of computer vision convoys several loss functions such as triplet [45], quadruplet [46], lifted structure [47], \(N\)-pairs [48], and angular [49] losses. In this work, we extend the softmax embedding variant mainly proposed for image-to-image retrieval to the case of text/image retrieval. Because of memory requirements, we learn iteratively these distances on small batches sampled from the complete dataset. The aim is to make the features of an image closest to its corresponding textual description and different from other image features and vise-versa.  
If we consider \(B_{k}=\{X_{i},Y_{i}\}_{i=1}^{m}\) as the \(k\) th mini-batch of size \(m\) sampled instances from the full dataset \(\mathcal{D}\), then for a sentence \(Y_{i}\), its corresponding image \(X_{i}\) should be classified into instance \(i\), while other images \(X_{j}\), with \(j\
eq i\) are not classified into instance \(i\). Therefore, the probability of the image \(X_{i}\) being recognized as instance \(i\) is defined by:  
\[P(i|X_{i})=\frac{\textit{exp}\Big{(}f^{T}(Y_{i})f(X_{i})/\tau\Big{)}}{\sum_{k=1 }^{m}\textit{exp}(f^{T}(Y_{k})f(X_{i})/\tau)} \tag{1}\]  
where \(\tau\) is a temperature parameter controlling the sharpness of the distribution. Similarly, the probability of a sentence \(Y_{j}\) not being assigned to instance \(i\) can be defined by \(1-P\Big{(}i\big{|}Y_{j}\Big{)}\) where  
\[P\Big{(}i|Y_{j}\Big{)}=\frac{\textit{exp}\Big{(}f^{T}(Y_{i})f\big{(}Y_{j} \big{)}/\tau\Big{)}}{\sum_{k=1}^{m}\textit{exp}\Big{(}f^{T}(Y_{k})f\big{(}X_{ j}\big{)}/\tau\Big{)}},\ k\
eq j \tag{2}\]Then under the assumption that different sentences being recognized as instance \(i\) are independent, the joint probability image \(X_{i}\) being recognized as instance \(i\) and the sentence \(Y_{j}\) not assigned to instance \(i\)  where \(j\
eq i\) is simply \(P(i|X_{i})\prod_{j\
eq i}\bigl{(}1-P\bigl{(}i\bigr{|}Y_{j}\bigr{)}\bigr{)}\).  
Then the corresponding negative log-likelihood for the min-batch \(B_{k}\) can be given as follows:  
\[J_{i}=-log(P(i|X_{i}))-\sum\
olimits_{j\
eq i}log\bigl{(}1-P\bigl{(}i\bigr{|} Y_{j}\bigr{)}\bigr{)} \tag{3}\]  
The corresponding total log-likelihood \(J\) over the entire dataset is:  
\[J_{sentence}=-\sum\
olimits_{i}log(P(i|X_{i}))-\sum\
olimits_{i}\sum \
olimits_{j\
eq i}log\bigl{(}1-P\bigl{(}i\bigr{|}Y_{j}\bigr{)}\bigr{)} \tag{4}\]  
Similarly, we can extend this formulation in the reverse direction that form image \(X_{i}\) to sentence \(Y_{i}\) yielding the following log-likelihood:  
\[J_{Image}=-\sum\
olimits_{i}log(P(i|Y_{i}))-\sum\
olimits_{i}\sum\
olimits_{ j\
eq i}log\bigl{(}1-P\bigl{(}i\bigr{|}X_{j}\bigr{)}\bigr{)} \tag{5}\]  
Then the total log-likelihood used as loss for learning the parameters of the proposed asymmetric Siamese network is given by:  
\[J=\lambda_{1}J_{sentence}+\lambda_{2}J_{Image} \tag{6}\]  
where \(\lambda_{1}\) and \(\lambda_{2}\) are balancing weights.