Question: How do they perform two passes over the collection when modeling equations using EqEmb?
Evidence: We analyzed EqEmb models performance using a held out set of words that we generate for each equation in our collections. Held out sets are constructed using the following procedure: We traverse over the collections and for every discovered equation we randomly sample words from its context set. The held out set contains the sampled words and their context window which also includes the equation. For each held out word we also generate a set of negative samples for the given word context. We perform the same procedure to form a validation set. For each of the INLINEFORM0 equations in a collection, two held out words INLINEFORM1 are sampled. For a context window of size 4 the sampled word context is defined as INLINEFORM2 .
During training we compute the predictive log-likelihood on the validation set of words using the fitted model after each iteration over the collection. A fitted model is a collection of interaction and feature vectors for each equation and word. Given a fitted model, the log probability of a held out word is computed using the following formula: DISPLAYFORM0 
which is the softmax function computed over a set of negative samples INLINEFORM0 and the held out word. In particular, we ran the model 20 times across the collection. After each collection iteration INLINEFORM1 we observe whether the predictive log-likelihood continues to improve compared to the previous ( INLINEFORM2 ) iteration. We stop at the INLINEFORM3 -th iteration when that is no longer the case.
When modeling equations using EqEmb we perform two passes over the collection. In the first pass we only model words while ignoring equations. In the second pass we only model equations while holding fixed the interaction and feature vectors of all words. In context of EqEmb we treat equations as singleton words and the broader question that we are trying to answer is whether we can learn something about the meaning of the singleton words given the fixed word interaction and feature vectors.
In our analysis we evaluated the performance of the EqEmb models across different sizes for the word context (W), word-equation context (E) and embedding vector size (K). Model performance was compared with 4 existing embedding models: b-emb, CBOW, GloVe and PV-DM. We used the gensim BIBREF13 implementation of the CBOW and PV-DM models. When modeling equations using the first 3 embedding models we treat equations as regular words in the collection. In case of the PV-DM model we parse the article so that equations and their surrounding context of length equivalent to the word-equation context window are labeled as a separate paragraph. We also assign paragraph labels to the article text occurring between equation paragraphs.
When modeling equations using EqEmb we perform two passes over the collection. In the first pass we only model words while ignoring equations. In the second pass we only model equations while holding fixed the interaction and feature vectors of all words. In context of EqEmb we treat equations as singleton words and the broader question that we are trying to answer is whether we can learn something about the meaning of the singleton words given the fixed word interaction and feature vectors.
Answer: They model words and equations separately in two passes, and hold fixed the interaction and feature vectors of all words when modeling equations.