# Multimodal Wireless Situational Awareness-Based Tourism Service Scene

Authors: Yan Liu; Shuo Zhu
Journal: Journal of Sensors
 (2021)

---

## Abstract

Community platforms featuring user sharing and self-expression in social media generate big data on tourism resources, which, if fully utilized in a smart tourism system driven by high-tech and new technologies, will bring new life to the field of smart tourism research and will play an important role in the development of Internet+ tourism. However, tourism data in social media has the following characteristics: diversity, redundancy, heterogeneity, and intelligence. To address the characteristics of tourism data in social media, this thesis focuses on the following challenges: it is difficult to efficiently obtain tourism visualization information (text and images) in social media; it is difficult to effectively utilize tourism multimodal heterogeneous information; it is difficult to properly retrieve multimedia entity information of tourism attractions; and it is difficult to reasonably construct tourism personalized recommendation models. In this paper, an image search reordering method based on a hybrid feature graph model is proposed to realize the rapid acquisition of high-quality Internet images from the web using hybrid visual features and graph models, thus providing data security for the analysis of social media-based tourism images. To address the shortcomings of current search engines for image retrieval, visual information is used to bridge the problem of semantic gap between text-based search and images. To address the limitation of single visual features, we use latent semantic analysis to fuse multiple visual features to obtain hybrid features, which not only combine multiple single features but also preserve the potential relationship between these features. To address the shortcomings of the reordering methods based on classification and clustering, a reordering framework based on the graph model is used to reorder the images and finally complete the image search reordering based on the hybrid feature graph model. This method can obtain image information in social media with high efficiency and quality and then prepare for the subsequent work of tourism image analysis mining and personalized recommendation.


---

## Body

## 1. Introduction

Information technology has entered the fast lane of development, and the Internet has entered the Web 3.0 era, which means that the era of “Internet+integrated services” has come [1]. During the “12th Five-Year Plan” period, China’s information technology sector has undergone radical changes, and the premier’s government work report has put forward the macro strategy of “Internet+,” which has made the Internet the most important basic platform widely used by all walks of life. The so-called “Internet+” is a mode of providing comprehensive services in the Web 3.0 era, which is a combination of the Internet and traditional industries. As the information technology of the Internet era and traditional social industries are constantly interacting and rapidly integrating, endlessly creative technologies and industries are emerging [2]. In the field of communication, with the emergence and development of instant communication, the Internet has promoted the continuous change and upgrade of operators’ related business; in the field of transportation, with the prevalence of travel software, the Internet has improved people’s travel mode, while increasing the efficiency of the use of socially owned vehicles and contributing to environmental protection; in the field of finance, with the popularity of online payment, Internet finance has received strong support from national policies and has begun to enter an orderly development stage [3, 4]. Thus, “Internet+” has played an irreplaceable role in promoting traditional industries and has given rise to a series of tertiary industries based on the Internet, such as Internet+ communication, Internet+ transportation, Internet+ finance, and Internet+ medical care [5].Therefore, in the era of Internet+integrated services, massive data processing, ubiquitous computing and analysis, and the new era of Internet concepts will eventually cause changes and innovations in traditional industries. The rapid development of the national economy has been accompanied by a significant improvement in people’s living standards and a stronger demand for travel, which has led to a boom in tourism [6]. The development of Internet+ tourism has changed the pattern of the traditional tourism seller’s market, and personalized free travel has become the trend of tourism development [7]. Data released by the National Tourism Administration in 2014 showed that for domestic travel, the number of tourists choosing travel agencies was only 3.6% of the total number of domestic trips; for foreign travel, the number of tourists choosing free travel was 65% of the total number of outbound trips [8]. This phenomenon is due to the fact that people have access to a huge amount of travel information on the Internet, which facilitates travel planning and decision-making, which in turn has led to a great change in the way people travel and their perceptions in recent years. As a result, there is a growing demand for high-quality, in-depth, intelligent travel information [9]. For example, people want more and more effective travel information and intelligent and personalized recommended attractions to help them make decisions and develop their itineraries [10]. In order to respond to the demand of travelers for personalized travel, the tourism industry urgently needs to make use of the powerful information integration function of the Internet and introduce intelligent personalized travel services for different people through continuous technological innovation and deep-seated industrial upgrading. The informatization of tourism industry is the combination of “Internet+tourism” and information technology [11]. Information technology has penetrated into all aspects of the tourism industry, from access to information about attractions and panoramic views of attractions to personalized travel needs, and the level of information construction of tourism companies has affected their position and competitiveness in the tourism industry [12]. Therefore, it is the development goal of the tourism industry to make the tourism industry penetrate deeply into the Internet, to use information technology throughout the tourism industry, and to realize the modernization, informatization, and internationalization of tourism. The informatization of tourism industry will open a new revolution in tourism industry and will give the prospect of Internet+tourism application with unlimited development space and great inherent potential, which is also the foundation stage of intelligent tourism. Tourism information is the basic data guarantee of tourism informatization, so it is especially important to obtain high-quality tourism data resources. Tourism information not only exists in large quantities in government tourism-related departments and tourism companies, but with the step-by-step development of social media in recent years, there also exists a huge amount of available tourism information resources on major social media websites [13]. However, it is difficult for people to accurately obtain data from the vast amount of travel information, especially for diverse travel destinations or the dazzling number of travel routes. The current tourism information service simply displays tourist attraction information on the Internet, and its shortcomings are mainly reflected in the following: (1) information is not comprehensive enough, not making full use of the rich tourism information shared on social media; (2) information is only passively presented to users, not being able to find and filter information according to user needs; and (3) ignoring user needs, not considering contextual information of user scenarios and not doing customized and personalized services for users.Without intelligently considering the comprehensiveness and preference of information, it often fails to recommend satisfactory results for users. These shortcomings have severely restricted a series of travel behaviors such as access to high-quality travel information, demand-compliant destination recommendations, and personalized plan customization. Therefore, in addition to mastering the huge tourism information resources, informatization of tourism industry also requires specialized processing, mining, and analysis of these massive data. From acquiring relevant travel information to sharing travel experiences and from appreciating precise information search to feeling personalized information recommendation, travelers have a huge demand for informatization and intelligent application of tourism industry. Therefore, the use of data mining and machine learning and other technologies to mine tourism information resources and realize intelligent tourism based on data mining is imminent. And the current application of the intelligent tourism system still has many problems, such as ignoring the massive information in the social media environment, the system intelligent function is not clear, the user needs are not paid attention to, etc., which makes the development of intelligent tourism based on tourism information mining seriously restricted. Tourism information application needs to both meet the needs of travelers to collect travel information and help them to make travel plans and decisions, thus greatly improving the travel experience and enhancing user satisfaction. Therefore, the tourism industry, focusing on the current development needs, urgently needs to comply with the development of Internet+tourism informatization, actively mining tourism data and information, striving to explore intelligent and personalized tourism, and eventually making continuous efforts to realize intelligent tourism.


## 2. Related Work

With the advent of Web 2.0, online multimedia sharing sites, such as Flickr, Tripadvisor, and YouTube, have become prevalent, and the information uploaded by users contains a large amount of travel-related content, which can be widely used in travel systems [14]. Therefore, in recent years, many intelligent travel systems have been built to make travel easier and faster by analyzing and mining travel multimedia information to achieve accurate search or personalized recommendation functions. Wikitravel is an early travel information system before the advent of Web 2.0, which provides users with open, complete, real-time, and trusted travel information. VirtualTour provides an online travel service dedicated to mining high-quality travel photos from image-sharing sites and designing a user interface with search and map location functions to help users plan their routes for travel. By analyzing over 110,000 geotagged images on Flickr, DiverseSearch uses images to generate a visual view of attractions while meeting the diversity of attraction search results. PersonalizedMM combines text, geotagged images, and videos to generate attraction summaries and then recommends personalized attraction summaries to users by query [15, 16].Photo2Trip takes 20 million images from Panoramio and 200,000 travelogues from the web, mines them for descriptions of attractions and routes between them, and then provides trip planning services for tourists. gTravel is a global social travel system that not only helps users plan and navigate their trips but also monitors their location to change their routes at any time [17]. In addition, users can also share their travel experiences or access other people’s travel experiences on the system platform. Through the above introduction of typical intelligent tourism systems, we can understand the current development status of intelligent tourism systems, which are characterized by obtaining tourism-related information from social media and then applying it to intelligent tourism systems to provide application services after analysis and mining [18]. In addition, the focus of information utilization varies from system to system. From the type of information utilized, most systems utilize rich tourism heterogeneous information, including text, image, and numerical information, while some personalized intelligent tourism systems incorporate user contextual information and some systems introduce user interaction to analyze user preferences [19].Intelligent tourism systems in which multimedia information about tourism is analyzed and mined and some kind of application is realized are mainly divided into two categories according to their application to tourism information: one type of system is concerned with the intrinsic representation of information. The researchers propose an online tourism system that is dedicated to finding high-quality images from photo-sharing sites and using them to represent the content of attractions. The researchers analyzed geotagged photos from the Flickr site and then mined the image-based multiperspective representations of attractions to ensure visual diversity in attraction image search results [20]. The researchers combine user queries and travel multimedia information obtained from social media to generate personalized summaries of attractions and recommend them to users. The researchers propose a new framework for image classification, which uses tagged attraction images to construct 3D visualization models and then classifies images into attraction categories to which they belong based on the popular images of the attraction. Another class of systems focuses on the problem of retrieving and recommending multimedia information for tourism. Photo2Trip uses the acquired images and travelogues to explore routes within and between attractions and then provides users with travel route planning. It is a social travel system that helps users plan their trips [21]. The researchers propose a mobile travel search framework that can show users multiple views of attractions based on image information through compressed transmission techniques. The researchers’ work processes low-resolution query images through a remote server, then identifies and searches for attractions, and reconstructs the corresponding attractions in a three-dimensional view based on a photo set. In the current state of development, the main concern of intelligent tourism systems is how to obtain high-quality information from the vast amount of tourism-related multimedia information in the web and how to apply this high-quality information in a rational way in intelligent tourism.


## 3. Multimodal Composition of Tourism Service Scenarios

### 3.1. Sensory Modality

For the query term “apple,” the relevance of the images obtained using shape features is higher, while for the query term “butterfly,” image search reordering using SIFT features yields images with higher relevance to the query term. The visual modality of the propaganda film refers to the discursive means of expressing Chongqing’s city image visually to the audience through the dynamic images, various shots, and the text that may appear in the images. As we can see in Figure1, the visual mode is the most important way of shaping and communicating the image of Chongqing in the text of Chongqing city image promotion film. In this mode, the film shows a variety of city image elements. For example, Chongqing’s first urban promotional film, “Infinite Chongqing,” presents Chongqing’s image in many aspects through visual modality: the city of mountains and water, the city of clouds and fog, the city of busyness and vitality, and the city of special humanities, shaping a city landscape and humanities with diversified characteristics.Figure 1
Example of reordering results based on query terms.Sound mode refers to the sound that can be heard in the Chongqing city image film, including the following: (1) the music that can highlight the theme of the film and the narration that introduces the historical development of Chongqing and (2) the sound around the shooting site recorded during the filming of the film, such as wind, water, birdsong, bicycle bells, airplanes, and light rail station announcements. The use of sound modalities can make the content of motion pictures more infectious and persuasive. For example, in the promotional film “This City, Chongqing,” time-lapse photography is used to show the image of Chongqing with the sound of bicycle bells, birdsong, light and dynamic music, wind chimes, and thunderstorms, showing the image of a city with a strong sense of life and bustling life.In order to make up for the shortcomings of current search engine algorithms, image search reordering based on visual features is needed, so the research problem of image search reordering based on visual features has attracted a lot of attention from researchers. The basic idea of visual feature-based image search reordering is to use visual features to reorder the images returned by the search engine and then return them to the user, hoping that the user can get the images related to the search query in the top images, and the ideal state of reordering is that the relevance of the top images is higher, saving the time of the user to find the images related to the query terms from the redundant results. The existing visual feature-based image search reordering improves the current search engine to some extent and increases the relevance of the top-ranked images. From the perspective of using visual features, image search reordering methods are mainly divided into two categories: global visual feature-based and local visual feature-based image reordering methods. The global visual feature-based ranking methods mainly utilize features such as color and texture, while the local visual feature-based reordering mainly utilizes features such as scale-invariant feature transform (SIFT). However, global and local features describe images from different perspectives and each has its own limitations, which means that reordering images returned for different query terms does not always yield satisfactory results if only global or local visual features are used. Figure1 shows a visual example to illustrate how image search reordering works better for images returned by different query terms with only specific types of visual features.This situation occurs because the images returned by search engines have different sensitivities to different types of visual features. Since reordering with a single type of visual feature cannot always achieve good reordering results, in order to improve the generality of image search reordering algorithms to overcome the problem that different images have different sensitivities to different visual features, it is necessary to combine multiple visual features for image search reordering to compensate for the limitations of single features. The methods of combining multiple visual features are generally divided into two categories: one is to use different visual features for reordering and then perform decision-level linear fusion of the reordering results and the other is to fuse the visual features and then perform reordering. Therefore, this section focuses on the research of methods for image reordering based on multivisual feature fusion to bridge the semantic gap between text and images. To perform feature fusion, this section proposes a hybrid feature based on latent spatial analysis (LSA) for reordering, which not only preserves the important information of both types of features in the process of dimensionality reduction but also mines the potential association information between features.


### 3.2. Multimodal-Based Search Reordering

To compensate for the limitations of search engines using text to retrieve images, visual features are used to reorder images to bridge the semantic gap between text and images, and because images have different sensitivities to different visual features, multiple visual features are fused to generate hybrid features for reordering. Meanwhile, to ensure the relevance of query terms and reordered images, a graph model-based reordering framework is used to complete the reordering of image search results, thus helping users to be able to obtain the most relevant images to their queries from the large number of images returned by search engines. The proposed hybrid feature graph model-based image search reordering method is to mix visual features and then use the graph model-based reordering framework to complete the reordering. Figure2 gives the block diagram of the hybrid feature graph model-based image search reordering method, which is mainly divided into two parts: learning hybrid features and graph-based reordering. In the offline learning phase, all the images are learned based on visual feature fusion using latent semantic analysis after extracting visual features. In the online ranking stage, a query word is given, and the initial ranking result of the returned images is obtained after matching with the textual information of the images, and then, the similarity between the images based on the hybrid features is calculated under the framework of the reordering of the graph model to complete the construction of the graph model, and finally, the reordering result is given.Figure 2
Block diagram of image search reordering based on the hybrid feature map model.Since search engines do not always return satisfactory image results based on query terms, users only focus on images that are ranked at the top. From the image reordering rules, it is clear that two assumptions need to be satisfied in constructing the graph model-based reordering, namely, visual uniformity and initial ranking union. Therefore, the image search reordering results are obtained by solving the optimization problem based on the graph model and satisfying the reordering assumptions. Given a query term, a graph is constructed using the images returned by the search, where each image represents a node in the graph, and the images are connected with edges whose weights represent the similarity between images based on hybrid features.


### 3.3. Scene Feature Fusion

The basic idea of visual feature-based image search reordering is to use visual features to reorder the images returned by the search engine and then return them to the user, hoping that the user can get the images related to the search query in the top images, and the ideal state of reordering is that the relevance of the top images is higher, saving the time of the user to find the images related to the query terms from the redundant results. First, the images in the dataset need to be preprocessed; i.e., the image specifications and formats are unified. Then, the images are subjected to feature extraction, and since BovW features and color moment features, which represent visual features, describe the images in terms of local details and colors, respectively, and both are visually representative, BovW features and color moment features are selected for image feature fusion in this section. For the BovW features, that is, the feature vectors are extracted from the images using the visual bag-of-words model. First, the scale rotation invariance (SIFT) features are extracted from each image, where the dimension of SIFT features is 128.(1)Nt−1=∑i=0nϕCs,t,t+Bt,where N referred to the feature vectors and t means the scaling time.Then, the SIFT features are clustered using theK-means method, the cluster center is considered the visual word of the image, the distance from the SIFT feature point to the cluster center is calculated for each image, and the SIFT features are mapped to the visual word in its cluster center.
(2)Lty=C−t1−tlχ1+tc,where C means the cluster center and L means the feature points.Finally, the image is represented as a feature vector based on the visual words. For the color moment features, since the image can express the color distribution using the low-order moments of color (first-order, second-order, and third-order) on all three color channels, the image representation is performed on the low-order moments, where the dimension of the color moments is 255.(3)yix,y=pit−1pit−δ.The basic idea of potential semantic space analysis is to map the high-dimensional visual feature space to the low-dimensional visual vector space for representation. Here, Singular Value Decomposition (SVD) is used to solve the latent semantic space of visual features.(4)Ut=∂C∂t+αTβN,Kt=t+1−αKt−1,where U means the visual feature space and Kt can be referred to as the visual vector space.For the query terms in the MSRA-MM database, the top 400 images returned by the search results for each query term are reordered, and then, the relevance of the reordered images to the query terms is evaluated. Since users will give preference to the images sorted in front, the higher the relevance of the previous images, the better the sorting effect, as shown in Figure3. Therefore, based on the principle that images with high relevance need to be ranked higher after reordering, the performance of the proposed image reordering method is evaluated using Normalized Discounted Cumulated Gain (NDCG), i.e., the relevance of the ranked position of each image to the query term. For the evaluation of the image search reordering method proposed in this section, CrowdReranking, which overcomes the limitation of a single search engine to return images from multiple search engines, is a classical method of reordering based on clustering, where relevant concepts are found from the initially sorted images to pretrain concept detectors, and then, clustering is performed to complete the reordering. CrowdReranking can mine relevant visual prototypes from the query results returned by multiple search engines to complete the reordering of mixed results from multiple search engines.Figure 3
Reordering of the graph model.


## 3.1. Sensory Modality

For the query term “apple,” the relevance of the images obtained using shape features is higher, while for the query term “butterfly,” image search reordering using SIFT features yields images with higher relevance to the query term. The visual modality of the propaganda film refers to the discursive means of expressing Chongqing’s city image visually to the audience through the dynamic images, various shots, and the text that may appear in the images. As we can see in Figure1, the visual mode is the most important way of shaping and communicating the image of Chongqing in the text of Chongqing city image promotion film. In this mode, the film shows a variety of city image elements. For example, Chongqing’s first urban promotional film, “Infinite Chongqing,” presents Chongqing’s image in many aspects through visual modality: the city of mountains and water, the city of clouds and fog, the city of busyness and vitality, and the city of special humanities, shaping a city landscape and humanities with diversified characteristics.Figure 1
Example of reordering results based on query terms.Sound mode refers to the sound that can be heard in the Chongqing city image film, including the following: (1) the music that can highlight the theme of the film and the narration that introduces the historical development of Chongqing and (2) the sound around the shooting site recorded during the filming of the film, such as wind, water, birdsong, bicycle bells, airplanes, and light rail station announcements. The use of sound modalities can make the content of motion pictures more infectious and persuasive. For example, in the promotional film “This City, Chongqing,” time-lapse photography is used to show the image of Chongqing with the sound of bicycle bells, birdsong, light and dynamic music, wind chimes, and thunderstorms, showing the image of a city with a strong sense of life and bustling life.In order to make up for the shortcomings of current search engine algorithms, image search reordering based on visual features is needed, so the research problem of image search reordering based on visual features has attracted a lot of attention from researchers. The basic idea of visual feature-based image search reordering is to use visual features to reorder the images returned by the search engine and then return them to the user, hoping that the user can get the images related to the search query in the top images, and the ideal state of reordering is that the relevance of the top images is higher, saving the time of the user to find the images related to the query terms from the redundant results. The existing visual feature-based image search reordering improves the current search engine to some extent and increases the relevance of the top-ranked images. From the perspective of using visual features, image search reordering methods are mainly divided into two categories: global visual feature-based and local visual feature-based image reordering methods. The global visual feature-based ranking methods mainly utilize features such as color and texture, while the local visual feature-based reordering mainly utilizes features such as scale-invariant feature transform (SIFT). However, global and local features describe images from different perspectives and each has its own limitations, which means that reordering images returned for different query terms does not always yield satisfactory results if only global or local visual features are used. Figure1 shows a visual example to illustrate how image search reordering works better for images returned by different query terms with only specific types of visual features.This situation occurs because the images returned by search engines have different sensitivities to different types of visual features. Since reordering with a single type of visual feature cannot always achieve good reordering results, in order to improve the generality of image search reordering algorithms to overcome the problem that different images have different sensitivities to different visual features, it is necessary to combine multiple visual features for image search reordering to compensate for the limitations of single features. The methods of combining multiple visual features are generally divided into two categories: one is to use different visual features for reordering and then perform decision-level linear fusion of the reordering results and the other is to fuse the visual features and then perform reordering. Therefore, this section focuses on the research of methods for image reordering based on multivisual feature fusion to bridge the semantic gap between text and images. To perform feature fusion, this section proposes a hybrid feature based on latent spatial analysis (LSA) for reordering, which not only preserves the important information of both types of features in the process of dimensionality reduction but also mines the potential association information between features.


## 3.2. Multimodal-Based Search Reordering

To compensate for the limitations of search engines using text to retrieve images, visual features are used to reorder images to bridge the semantic gap between text and images, and because images have different sensitivities to different visual features, multiple visual features are fused to generate hybrid features for reordering. Meanwhile, to ensure the relevance of query terms and reordered images, a graph model-based reordering framework is used to complete the reordering of image search results, thus helping users to be able to obtain the most relevant images to their queries from the large number of images returned by search engines. The proposed hybrid feature graph model-based image search reordering method is to mix visual features and then use the graph model-based reordering framework to complete the reordering. Figure2 gives the block diagram of the hybrid feature graph model-based image search reordering method, which is mainly divided into two parts: learning hybrid features and graph-based reordering. In the offline learning phase, all the images are learned based on visual feature fusion using latent semantic analysis after extracting visual features. In the online ranking stage, a query word is given, and the initial ranking result of the returned images is obtained after matching with the textual information of the images, and then, the similarity between the images based on the hybrid features is calculated under the framework of the reordering of the graph model to complete the construction of the graph model, and finally, the reordering result is given.Figure 2
Block diagram of image search reordering based on the hybrid feature map model.Since search engines do not always return satisfactory image results based on query terms, users only focus on images that are ranked at the top. From the image reordering rules, it is clear that two assumptions need to be satisfied in constructing the graph model-based reordering, namely, visual uniformity and initial ranking union. Therefore, the image search reordering results are obtained by solving the optimization problem based on the graph model and satisfying the reordering assumptions. Given a query term, a graph is constructed using the images returned by the search, where each image represents a node in the graph, and the images are connected with edges whose weights represent the similarity between images based on hybrid features.


## 3.3. Scene Feature Fusion

The basic idea of visual feature-based image search reordering is to use visual features to reorder the images returned by the search engine and then return them to the user, hoping that the user can get the images related to the search query in the top images, and the ideal state of reordering is that the relevance of the top images is higher, saving the time of the user to find the images related to the query terms from the redundant results. First, the images in the dataset need to be preprocessed; i.e., the image specifications and formats are unified. Then, the images are subjected to feature extraction, and since BovW features and color moment features, which represent visual features, describe the images in terms of local details and colors, respectively, and both are visually representative, BovW features and color moment features are selected for image feature fusion in this section. For the BovW features, that is, the feature vectors are extracted from the images using the visual bag-of-words model. First, the scale rotation invariance (SIFT) features are extracted from each image, where the dimension of SIFT features is 128.(1)Nt−1=∑i=0nϕCs,t,t+Bt,where N referred to the feature vectors and t means the scaling time.Then, the SIFT features are clustered using theK-means method, the cluster center is considered the visual word of the image, the distance from the SIFT feature point to the cluster center is calculated for each image, and the SIFT features are mapped to the visual word in its cluster center.
(2)Lty=C−t1−tlχ1+tc,where C means the cluster center and L means the feature points.Finally, the image is represented as a feature vector based on the visual words. For the color moment features, since the image can express the color distribution using the low-order moments of color (first-order, second-order, and third-order) on all three color channels, the image representation is performed on the low-order moments, where the dimension of the color moments is 255.(3)yix,y=pit−1pit−δ.The basic idea of potential semantic space analysis is to map the high-dimensional visual feature space to the low-dimensional visual vector space for representation. Here, Singular Value Decomposition (SVD) is used to solve the latent semantic space of visual features.(4)Ut=∂C∂t+αTβN,Kt=t+1−αKt−1,where U means the visual feature space and Kt can be referred to as the visual vector space.For the query terms in the MSRA-MM database, the top 400 images returned by the search results for each query term are reordered, and then, the relevance of the reordered images to the query terms is evaluated. Since users will give preference to the images sorted in front, the higher the relevance of the previous images, the better the sorting effect, as shown in Figure3. Therefore, based on the principle that images with high relevance need to be ranked higher after reordering, the performance of the proposed image reordering method is evaluated using Normalized Discounted Cumulated Gain (NDCG), i.e., the relevance of the ranked position of each image to the query term. For the evaluation of the image search reordering method proposed in this section, CrowdReranking, which overcomes the limitation of a single search engine to return images from multiple search engines, is a classical method of reordering based on clustering, where relevant concepts are found from the initially sorted images to pretrain concept detectors, and then, clustering is performed to complete the reordering. CrowdReranking can mine relevant visual prototypes from the query results returned by multiple search engines to complete the reordering of mixed results from multiple search engines.Figure 3
Reordering of the graph model.


## 4. Results and Analysis

The robustness of the reordering method is evaluated by experimentally verifying the reordering effect of the reordering method on the images returned by different query terms. An experimental comparison is designed; i.e., the effectiveness of the proposed reordering method is evaluated by using the correlation between manually labeled images and query terms compared with the reordering results. Here, the top twenty query terms among the selected 73 query terms are selected for evaluation. Also, the relevance of the images in the database to the query terms is manually marked for relevance. Since in practical applications, users using search engines to obtain images are generally most concerned with the first 10 or so photos returned by the algorithm, the reordering effect is evaluated for the first ten images returned as a result; i.e., the value of NDCG@10 is calculated. Since the proposed reordering method combines BovW features and CM features for reordering and the comparison methods used are for single feature reordering, to ensure fairness, the BovW features and CM features are applied to the comparison method separately, and then, the NDCG values of the two features are averaged to obtain the mean value of NDCG to compare with the proposed method in this section. The experimental results are shown in Figure4. The NDCG@10 values show that the proposed reordering method has higher NDCG@10 values for different query terms and does not perform poorly for images returned by a certain query term. In other words, the proposed method can be used in the framework of graph-based reordering, combined with the idea of LSA-based feature fusion to obtain mixed features for graph model-based reordering, and good reordering results can be obtained. Since the IB reranking method is based on clustering, it is not easy to find the clustering center of the image for the diversity of images when the correlation of the returned results of some query terms is small, so it cannot show a better reordering effect for the returned images of some query terms. In conclusion, it can be seen from the figure that for different query terms, the reordering method performs different results, which is due to the different sensitivity of the reordering method to different features under different query terms, which affects the robustness of its algorithm. And after calculating the average value for different features, the proposed reordering method still outperforms the average value of NDCG@10 of the comparison methods, which indicates that the hybrid features based on potential semantic space have higher robustness.Figure 4
Identification results of travel service scenarios.The change of NDCG value is shown in Figure5, and the NDCG value is the average of the NDCG values of the 20 query terms used in the experiment. From the figure, we can see that the effectiveness of the proposed method is still higher than that of the comparison method, which again shows that the proposed method can guarantee the stronger relevance of the images with higher reordering. It can also be seen that the value of NDCG@num decreases as num becomes larger, which is due to the fact that after reordering, the images with low relevance are ranked lower, thus affecting the NDCG value, which is also in line with the requirement of reordering. The above two experiments, which verify the robustness of the image search reordering method based on the hybrid feature graph model and the relevance of the returned images, illustrate that the proposed feature fusion method can retain the important information of the features and tap the correlation information between the features and that the two assumptions for reordering can be satisfied under the graph-based reordering framework, which can ensure both visual uniformity and reordering with reference to the initial ranking results. The reordering is based on the graph-based reordering framework.Figure 5
Comparison of results of reordering methods.In learning the hybrid features, the low-dimensional representation is performed while fusing the two features using the idea of latent semantic analysis to map the visual features from the high-dimensional feature space to the hybrid features in the low-dimensional vector space, and the setting of the dimensionalityk of the hybrid features is discussed. Since the dimensionality of BovW and color moments are 2000 and 255, respectively, the dimensionality of the reduced hybrid features should be smaller than the smallest feature dimension, i.e., k≤255. In order to better preserve the important information of the matrix, the value of k should not be too small, which will affect the retention of information and the representation of potential associations. In learning the hybrid features, λ is the joint parameter in learning the shared matrix, and its value can affect the participation of two features. Therefore, based on empirical values, λ=0.15. μ is the regularization parameter in the graph-based reordering framework to balance the energy and penalty terms, and considering the importance of visual uniformity and initial ranking combination, μ=0.5 based on empirical values. Figure 6 shows the initial ranking results and the reordering results based on color features, BovW features, and mixed features. The top ten returned images are listed for each type of feature, and the relevance of each image to the query term is labeled using a red bar chart, where all red is the most relevant, half red and white colors indicate relevance, and white represents irrelevance.Figure 6
Reordering results for different features.Figure7 shows that the query word “zoo” is given, and the initial sequence of results shows that several of the first ten images are not related to the visual characteristic of “zoo”; e.g., the sixth image is a scene of grass, which is ranked first because the tag text of this image contains the word “zoo,” but it is not relevant from the perspective of visual features. The second, fourth, fifth, and sixth images are all color-related, but not very relevant to the visual feature of “zoo.” The third column is the reordering based on BovW features, which is mainly related to local features, and the visual relevance of the reordered images is slightly better than the initial ranking results. The fourth column is the reordering based on mixed features, and it can be visually seen that the reordered images have a higher correlation than the other three sequences, and it is obvious that the third and fourth images in this column are also sorted together as much as possible to satisfy the assumption of visual uniformity.Figure 7
Scene keyword recognition.For the query term “Christmas” given in Figure8, it can be seen that the returned images satisfy the relevance to a certain extent. Specifically, the reordered images using color features and BovW features improve the relevance of the returned images to a certain extent compared with the initial sorting results, while the visual-based relevance of the reordered images based on the hybrid features is intuitively the highest, again verifying that the proposed hybrid features can well solve the need for visual feature-based reordering, which not only retains the important information of single visual features but also exploits the potential correlation information between visual features. The proposed hybrid features not only retain the important information of single visual features but also explore the potential correlation information among visual features. Therefore, the visual example shown in the figure illustrates again the effectiveness of the proposed hybrid feature graph model-based image search reordering method in this section. The reordering results based on different features are shown in the framework of reordering based on the graph model with the same set of parameters to ensure fair comparison.Figure 8
Feature classification of scenario keywords.The above experiments show that the proposed personalized recommendation algorithm can obtain good recommendation results based on the collective wisdom information of tourism. The preference-based attraction ranking method only shows the most popular and famous attractions but ignores users’ individual needs. Therefore, the preference-based attraction ranking method only recommends attractions of universal value and does not satisfy users’ individual preferences. The recommended attractions for users need to consider not only the accuracy and surprise but also the freshness of the recommended results to users. In order to improve the freshness of the user, the recommendation algorithm should consider the contextual information of the user. There are various kinds of contextual information, such as interest preference, emotion, time, and location. For example, we can get the user’s location to recommend attractions that match the user’s context. From a psychological point of view, most users tend to choose attractions that are closer to them. So when recommending attractions for users, contextual information will be used to correct the candidate attractions. In the proposed recommendation algorithm, the personalized attraction similarity model plays an important role in the recommendation, but in reality, the situational context affects the user’s travel decision, so the situational context information needs to be combined in the recommendation to match the real user psychology.


## 5. Conclusion

Internet+travel applications have become the biggest driving engine of the tourism industry, driving its rapid development. Along with the maturity and development of social media platforms, people can freely share what they see and hear on social media, and the massive travel information spread on social media has become the main source of travel information for people. It is a meaningful research topic to help people dig out valuable travel information from the massive social media data and explore the travel recommendation under Internet+travel application. In this paper, based on the characteristics of redundancy, diversity, heterogeneity, and intelligence of social media travel information, we use data mining methods to obtain useful and high-quality information for personalized travel recommendation. Therefore, this paper focuses on how to obtain high-quality visualized travel information from social media and analyze and personalize the social media travel information for recommendation, so that users can quickly get the required travel information from the massive social media travel information. Due to the shortcomings of current image search engines, there is a semantic gap between text and images for query word-based image search, so the images returned by the search need to be reordered for visual features. Since different single-modal visual information has limitations for images returned by different query terms, multiple visual features are fused to solve the one sidedness of single-modal visual information to represent images. And for simple multivisual feature fusion, it does not deeply mine the interrelationship of the visual features to be fused, so LSA-based multivisual feature fusion is used to learn hybrid features, which not only retains the important information of single visual features but also mines the association information of two visual features. The hybrid features are then used to perform image search reordering on the images returned from the initial search under the framework of graph model-based search reordering. In the future, the personalized attraction similarity model combined with contextual information outperforms the personalized attraction similarity model in practice and meets the user’s demand for freshness of the recommended attractions.



---

*Source: 5503333-2021-12-22.xml*