# A Review of Text Corpus-Based Tourism Big Data Mining
## 3 Text Corpus-Based Tourism Big Data Mining Techniques  
### Text Representations  
Recently, text data mining techniques have been mainly based on machine learning and deep learning, which play a decisive role for the improvement of NLP. In order to transform NLP problems into the problem of machine learning, symbols such as text need to be digitized firstly; that is, text representation must be obtained. In 2003, [PERSON] et al. [41] proposed a word vector model based on N-gram statistical language, which pioneered the neural network as a language model. Since then, a lot of word representations in low-dimensional space have been proposed, which are pre-trained on a large set of unlabeled text corpus. In contrast to high-dimensional space, these word representations or word embeddings can be compared in sematic distance and can be easily applied to other models. Traditionally, word embedding methods such as Word2 vec [26; 27] and Glove [28] were proposed to create a global word representation which considers the word in all sentences. Currently, more and more works have started to notice the different semantics of a word in different contexts. For example, contextual word vectors (CoVe) [29] capture contextual information by an encoder in an attentional seq-to-seq machine translation model; and embeddings from language model (ELMO) [30] extracts context-sensitive features from a bidirectional language model (biLM). Subsequently, with the proposal of Transformer network, generative pre-training Transformer (OpenAI-GPT) [31] and large-scale pre-training language model based on bidirectional Transformer (BERT) [32] pre-trained language models from Transformers for extracting contextual word embeddings and showed better performance than ever on many tasks [32]. Table 2 provides a directory of existing main pre-trained models which can be used for generating embeddings as an input of the next model training. In addition, the supervised methods for generating word vectors [42; 43; 44] can also promote word representations, such as the methods for improving the topics extraction or sentiment classification and so on. \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt}} \hline \hline
**Include** & **Exclude** \\ \hline Studies focused on text mining techniques based on macro corpus analysis, including topic extraction, text classification, sentiment classification, and text clustering. & NLP based on language scenes which requires knowledge rather than text, such as knowledge of domains and common sense. \\ \hline Tourism related studies in which text is the main research object, and other data structures can be the auxiliary means. & Tourism related studies in which other data structures (pictures, videos, etc.) are the main research object, while text can be the auxiliary means. Text corpus-based tourism big data mining related to tourist profiling and market supervision. & Tourism related studies in which other data structures (pictures, videos, etc.) are the main research object, while text can be the auxiliary means. Text corpus-based tourism big data mining related to question answering, or others. \\ \hline English texts. & All other languages. \\ \hline Studies published between 2014 and 2019. & Studies published before 2014. \\ \hline Peer reviewed academic journals or books, conference proceedings. & Dissertations, non-peer reviewed sources. \\ \hline In person context. & Online context. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Inclusion criteria. A word is a basic unit of a sentence, paragraph, or document. To better represent a text, converting the word vectors into a short text or long text representation is an efficient operation. Early typical text representations have bag-of-words (BOW) and term frequency-inverse document frequency (TF-IDF) models, but these document vector models are usually too simple and lack context information and word-to-word associations. They often perform poorly on some complex tasks.