NLP & Numeracy (Pt 2) — Embeddings, Language Models and Calculators
By: Mihail Dungarov
Tags: Naturallanguageprocessing, Data Science, Aritificial Intelligence, Bert, Machine Learning

NLP & Numeracy (Pt 2) — Embeddings, Language Models and Calculators

We explore how NLP equivalents of traditional calculators work. Machines that allow up to pose problems as text and retrieve results as text. Diagram and stick figure man by the author, beautiful icons by Freepik

In the previous article we took a look at what numeracy in natural language looks like and how it can impact NLP applications. We found different types of numeracy to pose different challenges for algorithmic processing and require different ML solutions. In this part, I would like to dive deeper into the ‘how’ of the different approaches.

We explore how NLP equivalents of traditional calculators work (see picture above). Machines that allow us to pose problems as text and retrieve results as text. Traditionally, when using a calculator the flow is as follows:

A human would formulate the input algebraically

algebraically A human would understand the input numerals and punch them into the machine

numerals and punch them into the machine A human would choose the operation to be performed

to be performed The calculator will then perform the operation

The calculator will provide the outputs in a structured way

Although I am talking about calculators, this flow applies without loss of generality from abaci to big data distributed systems. Accuracy is generally 100%, barring from rare errors due to data corruption.

The systems we are looking at in this and the previous article offer a slightly different flow to the above but since the core purpose is the same, I like to refer to them as NLP calculators.

With NLP Calculators, the flow is different in the sense that now:

Humans can provide the inputs in natural text

in natural text The NLP calculator, understands the inputs, chooses & performs the operation

Outputs are returned in natural language where appropriate

As we have seen in the previous article accuracy widely varies by approach and type of natural language request, however, with the most recent techniques, F1 is just above 90%, with human performance at ca. 96%.

Similar to the previous article, this one is a literature review with a focus on grouping works by their applications and approach. The rest of the article is organised as follows

Numeracy and Embeddings — will address what levels of numeracy can be expected at the embeddings level and what that means for dealing with numeric operations

— will address what levels of numeracy can be expected at the embeddings level and what that means for dealing with numeric operations Language Models and Numeracy — available approaches to better understand numeracy at language model level. Apart from making us better set up for numerically challenging downstream tasks, these can be applied to fact checking and error correction in text where the focus is numeracy.

— available approaches to better understand numeracy at language model level. Apart from making us better set up for numerically challenging downstream tasks, these can be applied to fact checking and error correction in text where the focus is numeracy. Calculators for Natural Language — these models are specifically developed for natural language question answering where numerical reasoning is needed and they end up being quite good at it

Numeracy of Embeddings

[Wallace, et al., 2019] set out to explore how much numerical awareness is naturally present within embeddings. Across most common embedding techniques, basic numeracy skill seems to be present. By using a wide range of commonly used embeddings (Word2Vec, GloVe, ELMo, BERT) and training a MLP (multi-layer perceptrons) to understand number values, list maxima/minima and perform addition, they show that the embeddings already contain significant numerical awareness. For instance, decoding (“three” → 3) and addition for numbers up to 1000 show about 1–2% RMSE (under 20 for values between 0–999)or less for embedding approaches mentioned abooo,with the exception of BERT which lags the rest. Models are trained on numbers in a certain range (e.g. -500 to +500) and then are asked about unseen numbers from that range, however, the authors note deterioration in performance when asked to work with numbers outside of that range. [Naik, et al. 2019] explore the same topic and confirm that embeddings successfully capture magnitude, however, get mixed results on numeration matching a string to a numeric value. However, their experiments focus on cosine distances between numeric embeddings, not the MLP representation what [Wallace, et al., 2019] allow for. This limits significantly the representations of numeracy allowed, potentially causing poorer performance.

Embeddings seem to be aware of number ordering and value, hence can account for the first numerical reasoning step above. Diagram and stick figure man by the author, beautiful icons by Freepik

Notably, these findings are about the properties of embeddings and do not clarify whether models can reason successfully numerically over text. In the diagram from the beginning of the article, this is equivalent to being able to identify which numbers are represented in the text. We still need a means by which to choose and perform the appropriate operation.

Language Models Numeracy

Language models (LM) aim to develop the ability to predict the next word in a sentence, a missing word from a sentence or a next sentence altogether (here we consider the case when predicting a missing word). A common metric for language models is perplexity, which measures how likely the correct answer is, according to the model.

Perplexity in Language Models

Every time a language model is asked to fill a missing word in a sentence, it will assign a probability to each of the possible choices (if it has a vocabulary of 300k words, it will produce 300k probability values.) For a good model, when predicting a missing word, most words will be considered very unlikely. In most cases, however, the actual correct word might not be the most likely one according to the model, because in most cases there are many possible correct answers in language (consider: “I got on the [BLANK] to get to work”, if I say “bus” but the answer is “tube” is it that too bad? No. What I want to avoid, however, is that the model is not totally thrown by “tube” being the correct answer. Thus, in order to measure how good a model is at filling a slot, we want to see how “perplexed” it is by the correct word. In the example above, it is desirable to have high probabilities for values such as: bike, bus, car, tube, etc. and not very high ones for words like banana, snake, cat, etc. Although if you have even watched “My Neighbour Totoro” you might disagree with one of these…

Perplexity and Numeracy in LM

LMs can be used as foundation for other tasks, e.g. frameworks like BERT where an LM can be fine-tuned to do question-answering, translation, information-retrieval tasks, etc. IN the context of numeracy, several papers have tried to understand how much inherent numeracy is there in the model by performing different “probing” tasks.

The applications can be two-fold:

Semantic Error Correction (SEC): LMs can help us spot if a number is “out of place” to correct errors in description of events where numerals play a role. [Spithourakis, et al., 2016] proposes the application in healthcare: quoting that clinical data can have up 26% error rates — discrepancies between the text and actual readings taken from a patient

LMs can help us spot if a number is “out of place” to correct errors in description of events where numerals play a role. proposes the application in healthcare: quoting that clinical data can have up 26% error rates — discrepancies between the text and actual readings taken from a patient Injecting Numeracy into LMs: [Geva, et al., 2020] add numerical reasoning steps to LM pre-training allowing for a model to then be used to fine-tune task specific tools while having better numeracy than BERT.

Semantic Error Correction (SEC)

[Lin, et al., 2020] find out that transformer based language models believe birds have four legs. They explore the model’s ability to pick up numerical commonsense knowledge — think “how many”-type questions for natural objects (e.g. sides of cubes, wheels of trucks, legs of animals). They demonstrate that LMs struggle to assign the right number. The accuracy at model’s first attempt (@1) is under 50% without pretraining and under 60% after fine-tuning and distant supervision, performance improves by the third attempt (@3) but there are only so many digits available anyway. [Spithourakis, et al., 2018] find that using a hierarchical model to first distinguish between numerals and words (i.e. non-numeral tokens), and separate model to deal with numerals when identified reduces perplexity by 2 to 4 orders of magnitude depending on the dataset. Notice that this pre-dates the Transformer era so the underlying architecture is an RNN. The results are impressive as the perplexity of the best models is half of those that do not allow for numeracy in any way. [Chen, et al, 2019] create a financial news headline dataset. They train a model to try and predict the order of magnitude of missing numeric values in a headline. Seven categories are used — numerical ranges increasing by a factor of 10 from decimals to 10⁶ (0 to 1, 1 to 10, 10 to 100, etc) They achieve a micro F1 of 70–80% depending on the dataset.

Injecting numeracy into LMs

[Geva, et al., 2020] maintain the framework pre-trained language models serving as a foundation for fine-tuning to specific tasks by adding numerical pre-training steps. Two datasets are added for pre-training: numeric and text data respectively. Additionally, they add random shift (randomized shift of the input tokens to either side to improve positional invariance of the numeric inputs) and a masked LM objective when training the model (aiming to minimize the loss on masked tokens). They demonstrate that they can achieve an F1 of over 70% question-answering on numeracy tasks (DROP dataset), without loss of performance on general question-answering (SQuAD dataset) compared to BERT. Generating numerical natural language examples: the authors follow a framework by [Hosseini, et al, 2014] to generate sentences like: Bridget adopted 4 brown dogs, The king recruited 1337 Irish soldiers, etc

NLP Calculators

These are models designed to consume text with numerically relevant content and output results of the numerical procedures needed. Just like a calculator we will input a numerical question and receive a result of the appropriate operation. The tiny difference — the input and the output here would be text. These are set apart from the rest by their significantly improved numeracy capability — coming very close to or just above 90% F1. With the exception of BERT-Calculator [Andor et al., 2019] these are purely numeric Question Answering systems — i.e. they are only trained to answer numerical questions.

The typical architecture would be:

Embedder & Encoder layers — one of several common methods could be used here to encode the inputs: word embeddings or using pre-trained transformer models — BERT , Roberta , Albert , etc. We saw in the previous sections that this step encodes information about magnitude, scale, etc of numbers

, , etc. We saw in the previous sections that this step encodes information about magnitude, scale, etc of numbers (Self-)Attention layers — these are essentially fine-tuned to the numeracy task. They perform steps A — C of the above diagram. Sometimes these are also explicitly separated — a dedicated module is trained to choose an operation (equivalent to step A above) and a set of sub-modules specialize in different numerical operations (addition, multiplication, sorting, etc) and provide an answer (steps B & C above). ISome of the earlier models, have dedicated output layers per numerical operation (eg. MTMSN [Hu, et al. 2019], NAQANet [Dua, et al., 2019]), In other cases, this is done by using the numerals and entities in a graph representation of for each of the question and paragraph (eg. NumNet [Ran, et al., 2019], QDGAT [Chen, et al., 2020]).

[Dua, et al., 2019] introduce the DROP benchmark as well as NAQANet. The model follows the format of QANet [Yu, et al., 2018] which is a model for traditional question-answering a la SQuaD. It differs by having 4 output layers, dedicated to different numeric tasks: counting, arithmetic, span (dealing with cases where a chunk of the paragraph is the answer) and multi-span (cases where several chunks of the paragraph are the answer). Finally, an Answer predictor is trained on the same attention layers as the four output layers to determine which of the four is the right type of answer. Leaderboard (LB) score — 64% F1. MTMSN [Hu, et al. 2019] extends this by, swapping the self-attention layers with full transformer blocks, adding negation to the possible tools as well as an answer predictor that also predicts the number of answers to specifically improve multi-spans and get holistic answers. Leaderboard (LB) score — 79.99% F1.

Graph representations / Graph NNs

NumNet [Ran, et al., 2019] also starts by first encoding the question and paragraph pair with pre-trained embeddings and ends with a prediction module, consisting of several predefined computational modules to provide the required answer. However, between the encode and prediction modules they introduce a “Reasoning Module” which creates a question and paragraph graph representation of the numbers in the inputs (picture below), where the numbers are the nodes and edges are “numerically aware” — aware of the greater than / less then relative scales of number nodes. A Graphical neutral network is used to reason over these relationships and feed into the output layers. NumNet+ follows a similar approach but uses Roberta (a version of BERT with robust hyperparameter choices RoBERTa) in the encoder and reasoning layer instead. Several NumNet+ are available on the LB with the best scoring F1 86%

NumNet [Ran, et al., 2019] introduce a “Reasoning Module” which creates a question and paragraph graph representation of the numbers in the inputs, where the numbers are the nodes and edges are “numerically aware” — aware of the greater than / less then relative scales of number nodes.

Some specialists

BERT+Calculator [Andor et al., 2019] aims to add numeracy in solving QA type tasks without loss of the traditional ability to answer factoid questions. The authors “give BERT a calculator” — a set of modules of the form: Operation(args,…) the model can then choose to “pick up” the calculator if needed to answer a numeric question. Crucially, BERT can also “choose” not to pick up the calculator and continue to answer factoid questions as before. BERT-Calculator (Ensemble) shows F1 81.8% on LB

[Andor et al., 2019] aims to add numeracy in solving QA type tasks without loss of the traditional ability to answer factoid questions. The authors “give BERT a calculator” — a set of modules of the form: Operation(args,…) the model can then choose to “pick up” the calculator if needed to answer a numeric question. Crucially, BERT can also “choose” not to pick up the calculator and continue to answer factoid questions as before. BERT-Calculator (Ensemble) shows on LB NeRd [Chen, et al., 2020] (I love the name, btw) generalises the approach taken by the above (e.g. NAQANet) by introducing a generic “Programmer” layer (LSTM layer). Unlike its predecessors, the “Programmer” has flexibility to learn useful computational patterns to generate an answer, as opposed to having a fixed set of operations available. Unlike NAQANet, The focus is on multi-span queries where most of the rest of the group significantly underperform. The NeRd (sorry, couldn’t resist) improves F1 by 15 points against its next contender at the time: MTMSN [Hu, et al. 2019] NeRd shows F1 81.7% on LB.

[Chen, et al., 2020] (I love the name, btw) generalises the approach taken by the above (e.g. NAQANet) by introducing a generic “Programmer” layer (LSTM layer). Unlike its predecessors, the “Programmer” has flexibility to learn useful computational patterns to generate an answer, as opposed to having a fixed set of operations available. Unlike NAQANet, The focus is on multi-span queries where most of the rest of the group significantly underperform. The NeRd (sorry, couldn’t resist) improves F1 by 15 points against its next contender at the time: MTMSN [Hu, et al. 2019] NeRd shows on LB. TASE [Segal, et al., 2020] improves on NeRd, by improving the ability to answer multi span questions. The improvement is due to treating the multi-span prediction task as a sequence tagging tasks as opposed to other models which predict each part of the answers individually, TASE — RoBERTa shows F1 83.62% on the LB

The Best for last

[Chen, et al., 2020] from the AntGroup introduce a family of models — QDGAT. The current leader on the DROP task is QDGAT-ALBERT (as of Dec 2020) a variation of QDGAT, achieving 90.1% F1 on the LB. They largely follow the NumNet architecture with significant extension of how Graph representations are handled, which also leads to significant improvement in performance. Graphs now take into account the types of numbers included in each sentence and the entity they relate to. Edge creation still represents the fact that two nodes coexist in a sentence. As before a Graph neutral network (GNN) is trained on those and different output layers are used for the different types of answers

QDGAT [Chen, et al., 2020] different color nodes represent the different number types and edges represent that sentences

The pictures shown throughout the articles are the DeepArt versions of the above QDGAT [Chen, et al., 2020] model architecture

Conclusion

The previous article gave an overview of the challenges that numeracy brings as well as how to measure it. Here we have seen how different approaches solve for that:

From [Wallace, et al., 2019] & [Naik, et al., 2019] we saw that basic understanding of size and magnitude is available in most commonly used embeddings. Thus we know that the ability to identify numbers is present in embeddings. Some limitations are seen as we can only find approximate numbers and even that loses accuracy outside of the training range

& we saw that basic understanding of size and magnitude is available in most commonly used embeddings. Thus we know that the ability to identify numbers is present in embeddings. Some limitations are seen as we can only find approximate numbers and even that loses accuracy outside of the training range We can train language models (LMs) to be numerically aware. [Spithourakis, et al., 2018] shows that by allowing a model to distinguish numerals can reduce perplexity of LMs by 2 to 4 orders of magnitude. GenBERT [Geva, et al., 2020] injects numeracy skills into BERT and maintains the ability to answer general natural questions (performance equals that of BERT on SQuAD) while achieving a 72% F1 on numerical questions (DROP data) vs BERT’s 33% F1.

shows that by allowing a model to distinguish numerals can reduce perplexity of LMs by 2 to 4 orders of magnitude. GenBERT injects numeracy skills into BERT and maintains the ability to answer general natural questions (performance equals that of BERT on SQuAD) while achieving a 72% F1 on numerical questions (DROP data) vs BERT’s 33% F1. With models like [Chen, et al., 2020] QDGAT we can answer numerical questions with F1 of over 90%, coming close to the human benchmark of 95% on the DROP dataset

While I will not trust these models blindly, I can at least feel more comfortable that there are a number of possible solutions when accurate NLP numeracy is needed.

Thank you for reading