Question: How to train domain-specific word embeddings?
Evidence: First, to operationalize, we say that term $i$ is associated with gender $j$ if, when discussing individuals of gender $j$, $i$ is used with unusual frequency – which we can check with statistical hypothesis tests. Let $f_i$ represent the likelihood of $i$ appearing when discussing women or men. $f_i$ is unknown, but we can model the distribution of all possible $f_i$ using the corpus of texts that we have from the domain. We construct a gender-balanced version of the corpus by randomly undersampling the more prevalent gender until the proportions of each gender are equal. Assuming a non-informative prior distribution on $f_i$, the posterior distribution is Beta($k_i$, $N - k_i$), where $k_i$ is the count of $i$ in the gender-balanced corpus and $N$ is the total count of words in that corpus.
As BIBREF10 discuss, “the distribution of the gender-specific counts can be described by an integral over all possible $f_i$. This integral defines the Beta-Binomial distribution BIBREF29, and has a closed form solution.” We say that term $i$ is significantly associated with gender $j$ if the cumulative distribution at $k_{ij}$ (the count of $i$ in the $j$ portion of the gender-balanced corpus) is $p \le 0.05$. As in the original work, we apply the Bonferroni correction BIBREF30 for multiple comparisons because we are computing statistical tests for thousands of hypotheses.
First, we trained domain-specific word embeddings using the Word2Vec BIBREF33 CBOW model ($w \in R^{100}$). Then, we used k-means clustering to cluster the embeddings of the gender-associated words. Since k-means may converge at local optima, we ran the algorithm 50 times and kept the model with the lowest sum of squared errors.
To automatically label the clusters, we combined the grounded knowledge of WordNet BIBREF34 and context-sensitive strengths of domain-specific word embeddings. Our algorithm is similar to BIBREF28's approach, but we extend their method by introducing domain-specific word embeddings for clustering as well as a new technique for sense disambiguation. Given a cluster, our algorithm proceeds with the following three steps:
Sense disambiguation: The goal is to assign each cluster word to one of its WordNet synsets; let $S$ represent the collection of chosen synsets. We know that these words have been clustered in domain-specific embedding space, which means that in the context of the domain, these words are very close semantically. Thus, we choose $S^*$ that minimizes the total distance between its synsets.
Candidate label generation: In this step, we generate $L$, the set of possible cluster labels. Our approach is simple: we take the union of all hypernyms of the synsets in $S^*$.
Candidate label ranking: Here, we rank the synsets in $L$. We want labels that are as close to all of the synsets in $S^*$ as possible; thus, we score the candidate labels by the sum of their distances to each synset in $S^*$ and we rank them from least to most distance.
In steps 1 and 3, we use WordNet pathwise distance, but we encourage the exploration of other distance representations as well.
First, we trained domain-specific word embeddings using the Word2Vec (Mikolov et al., 2013) CBOW model (w ∈ R100).
Answer: The authors trained domain-specific word embeddings using the Word2Vec CBOW model (w ∈ R100).