# Learning Word Embeddings with Chi-Square Weights for Healthcare Tweet Classification
## 2 Related Work
Neural network language models (NNLM) [30] produce a distributed vector representation of a word, known as a word embedding. The neural language model utilizes the neural network model to predict the word from the words appearing ahead of it [30], thus words with similar context will be mapped to close vector locations. In 2013, [PERSON] et al. [9] used a three-layer neural network model to build word embeddings, to capture the semantic and syntactic regularities through the words in the context window of the target word. They proposed two models: the skip-gram and CBOW models. Both learn the vector representation of the word from the context in which the word resides. The skip-gram model trains the weights in the hidden layer and uses a softmax function to produce a probability of appearance in the context for every vocabulary word. Since it is very expensive to compute every word's probability in the corpus for every sample, [PERSON] et al. adopt two mechanisms to further reduce the computation: hierarchical softmax and negative sampling. While hierarchical softmax uses a fixed Huffman tree structure with leaves as words in the vocabulary, negative sampling only samples \(n\) negative examples instead of the full vocabulary. [PERSON] et al. extended the skip-gram model from [PERSON]'s work and generated multiple vector representations for each word in a probabilistic manner [31]. Researchers also incorporated syntactic information into neural language models. [PERSON] and [PERSON] [16] extended [PERSON] et al. [9]'s skip-gram model by replacing the linear context with an NLP dependency-based syntactic context. Their model reported further improvement than the original model in the word similarity task (WordSim353 [32]). In this work, we focus on an extension of the CBOW model. There are several existing works that also develop this line of research. [PERSON] et al. [33] develop a very simple and effective method, incorporating additional information, the part-of-speech tag attached to each word during training. However, they did not invent a brand new model; instead, they used the CBOW model from Word2 Vec. For example, for polysemy disambiguation to train the embedding of (banks, verb) in the sentence \"He banks at the bank\", the input of CBOW is (\"He\", pronoun), (\"at\", deposition), (\"the\", determiner), (\"bank\", noun). For sentiment disambiguation, words are labeled with both the part-of-speech tag and sentiment for adjectives. Similarly, [PERSON] et al. used part-of-speech information to weight the context window in the CBOW model [11]. They argue that in their learning algorithm the part-of-speech tags capture syntactic roles of words and encode inherently the syntactic relationships inside the word vector representation. However, the authors overlook the ultimate goal of the learned embeddings. The usage of the pre-defined syntactic rules to weight the context does not guarantee later success in the classification task in which the trained embeddings will be used. Statistical measures have long been used in natural language processing. In terminology extraction, a fundamental processing step to extract technical terms from domain-specific textual corpora before complex NLP tasks, statistical measures such as mutual information, log likelihood and \(t\)-test are used to rank and identify the candidate terms from the texts. [PERSON] et al. developed a weighted voting algorithm that incorporated five existing term recognition algorithms to recognize both single- and multi-word terms in the text [34]. Most of the five term recognition algorithms adopt both statistical measures and frequency-based measures to rank the terms. In this work, we propose using \(\chi^{2}\) to weight the context words according to the words' contribution to the classification task. There is existing work which also uses statistical measures in the vector space model. [PERSON] introduced a count-based vector space model.