# TDCMR: Triplet-Based Deep Cross-Modal Retrieval for Geo-Multimedia Data
## 1 Introduction
However, this work ignores the cross-modal hashing representation that can enhance the search efficiency significantly. The GMR-Tree-based search algorithm cannot be directly used for cross-modal hashing retrieval. Therefore, to address these two challenges, we try to improve the semantic similarity constraints to guide the deep neural networks and designed a hybrid geo-multimedia index to organize cross-modal hash codes. Last but not least, we introduce one detailed example to describe this problem more vividly as follows. Example 1: As illustrated in Figure 1, a user wants to buy a blue shoe from a nearby shop. Unfortunately, he does not know the brand and of the shoe, and it is hard for him to use some words to describe the features of this shoe. Obviously, it is hard for him to find suitable shoes from nearby shops with limited information. In this case, he can input an image, which describes this blue shoe, and his location information by his mobile as a spatial multimedia \(k\) nearest neighbor query to a geo-tagged multimedia data retrieval system. According to his requirement, this system will return a result set, which contains k geo-multimedia data meeting his requirements. This result set indicates the user which shops have this kind of shoe and are close to his location. **Our Method. ** To this end, this paper proposes a novel efficient cross-modal hashing approach, termed as **T**riplet-based **D**eep **C**ross-**M**odal **R**etrieval (**TDCMR**). Specifically, a two-branch deep neural network-based backbone is integrated in the TDCMR framework, which is used to learn abstract semantic concepts. Besides, an improved triplet distance constraint is designed to capture multiple high-level similarities to capture semantic relationship among heterogeneous multi-modal data. Thus, the integrating of deep representation learning with an enhanced triplet distance constraint improves the cross-modal semantic learning performance. In addition, to realize efficient search on large-scale geo-multimedia data, a novel index, termed as **T**DCMR-Quadtree is proposed. It is a geo-semantic hybrid index integrated quadtree with cross-modal hash codes, which utilizes both geographic information and semantic similarity to realize candidate set pruning. Based on the index, an efficient cross-modal nearest neighbor query algorithm is developed for geo-multimedia retrieval. **Contributions. ** The main contributions of our work are summarized as follows:  
* We propose a triplet-based deep cross-modal hashing framework, named Triplet-based Deep Cross-Modal Retrieval (TDCMR), which aims to extract deep sample features to alleviate the semantic gap through a triplet deep neural network unified feature learning and hash learning process. * We carefully apply the efficiency TDCMR algorithm to utilize the Quadtree to improve and enhance the search performance significantly, named TDCMR-Quadtree, which is a novel index framework and can improve the retrieval efficiency. * We have conducted extensive experiments on three common used benchmarks, and the results demonstrate that our proposed method achieves very high performance. **Roadmap. ** In the remainder of this paper, we review the previous researches in Section 2. In Section 3, we introduce the definition of geo-multimedia data \(k\) nearest neighbor query and the related notions. In Section 4, we introduce the Triplet-based Deep Cross-Modal Retrieval (TDCMR) framework and its implementation. In Section 5, we evaluate our method on two geo-multimedia datasets. Finally, we conclude this paper in Section 6.