Question: What about the performance of EqEmbs on models?
Evidence: Equations are an important part of scientific articles, but many existing machine learning methods do not easily handle them. They are challenging to work with because each is unique or nearly unique; most equations occur only once. An automatic understanding of equations, however, would significantly benefit methods for analyzing scientific literature. Useful representations of equations can help draw connections between articles, improve retrieval of scientific texts, and help create tools for exploring and navigating scientific literature.
In this paper we propose equation embeddings (EqEmb), an unsupervised approach for learning distributed representations of equations. The idea is to treat the equation as a "singleton word," one that appears once but that appears in the context of other words. The surrounding text of the equation—and in particular, the distributed representations of that text—provides the data we need to develop a useful representation of the equation.
Figure FIGREF1 illustrates our approach. On the left is an article snippet BIBREF0 . Highlighted in orange is an equation; in this example it represents a neural network layer. We note that this particular equation (in this form and with this notation) only occurs once in the collection of articles (from arXiv). The representations of the surrounding text, however, provide a meaningful context for the equation. Those words allow us to learn its embedding, specifically as a "word" which appears in the context of its surroundings. The resulting representation, when compared to other equations' representations and word representations, helps find both related equations and related words. These are illustrated on the right.
EqEmbs build on exponential family embeddings BIBREF1 to include equations as singleton observations and to model equation elements such as variables, symbols and operators. Exponential family embeddings, like all embedding methods, define a context of each word. In our initial EqEmb, the context for the words is a small window, such as four or eight words, but the context of an equation is a larger window, such as sixteen words. Using these two types of contexts together finds meaningful representations of words and equations. In the next EqEmb, which builds on the first, we consider equations to be sentences consisting of equation units, i.e., variables, symbols, and operators. Equation units help model equations across two types of context—over the surrounding units and over the surrounding words.
We studied EqEmbs on four collections of scientific articles from the arXiv, covering four computer science domains: natural language processing (NLP), information retrieval (IR), artificial intelligence (AI) and machine learning (ML). We found that EqEmbs provide more efficient modeling than existing word embedding methods. We further carried out an exploratory analysis of a large set of INLINEFORM0 87k equations. We found that EqEmbs provide better models when compared to existing word embedding approaches. EqEmbs also provide coherent semantic representations of equations and can capture semantic similarity to other equations and to words.
We found that EqEmbs provide better models when compared to existing word embedding approaches.
Answer: EqEmbs provide better models when compared to existing word embedding approaches.