Neural Language Models | by Arun Jagota | Towards Data Science

In NLP, a language model is a probability distribution over sequences on an alphabet of tokens. A central problem in language modeling is to learn a language model from examples, such as a model of English sentences from a training set of sentences.
Language models have many uses. Such as...
Suggest auto-completes: the user types a few characters (or words) on a web search engine or a smartphone and likely extensions are suggested.
Recognize handwriting: An image-based recognition system augmented with a language model has improved accuracy. For instance, it can guess a word from its context even when it is not legible.
Recognize speech: As in handwriting recognition, augmenting a speech system with a language model improves its accuracy.
Detect and correct spelling errors
Translate languages: To translate text from one language to another, it helps to have language models for the languages.
In [1], we discussed statistical language models, a standard paradigm in the 1990s and earlier. In the past few decades, a new paradigm has emerged: neural language models. This is the subject of this post.
Why neural language models over statistical ones? The short answer: they yield state-of-the-art accuracy with minimal human engineering. Why? The short answer is: due to a combination of two breakthroughs in recent decades — deep neural networks, and recurrent ones. (Exponential increases in computing power and availability of training data also helped tremendously.) Distributed representations, i.e. word embeddings, also played a role.
In this post, we will cover recurrent neural networks (including deep ones) in the context of language modeling.
We start with the master equations. These are the most important take-away equations in this post. Beyond defining the language modeling problem precisely, they also serve to frame various approaches to its solution.
Let P(w1, w2, ..., wn) denote the probability of a sequence (w1, w2, ..., wn) of tokens. The aim of learning is to learn such probabilities from a training set of sequences.
Consider the related problem of predicting the next token of a sequence. We model this as P(wn | w1, w2, ..., wn-1). Note that if we can accurately predict the probabilities P(wn | w1, w2, ..., wn-1), we can chain them together to get
P(w1, w2, ..., wn) = P(w1)*P(w2 | w1)*P(w3 | w1, w2)* ... *P(wn | w1, w2, ..., wn-1)
Most learning approaches target language modeling as the next token prediction problem. This is a problem of supervised learning. The input is (w1, w2, ..., wn-1). The output we seek is a probability distribution over the various values that wn could take.
Note that the output is a fixed-length probability vector. (The dimensionality could be very high, though. Hundreds of thousands to millions for a language model on English sentences.)
The input is not a fixed-length vector. Its length, n, varies. In many use cases, n can often be large, in the 100s or even 1000s. This is the primary factor that makes this problem difficult.
Statistical approaches to estimating P(wn | w1, w2, ..., wn-1) when n is large must make some assumptions. Without them the problem is intractable. Consider n = 101 and say the alphabet just has two symbols: 0 and 1. The minimal training set size to reliably estimate P(wn | w1, w2, ..., wn-1) is of the order 2100. This is because the input can be any one of those choices, and we want to be able to predict the next token reasonably in each case.
The most widely-made assumption in statistical approaches is Markovian. Specifically, P(wn | w1, w2, ..., wn-1) is approximated by P(wn | wn-k, ..., wn-1). This assumes that the next token probabilities depend only on the last k tokens. Here k is fixed, often to a small number (2 or 3). See [1] for more on this topic.
This vastly simplifies the problem. At the cost of not being able to model long-range influences. Here is an example, from [2], of a long-range influence.
“Jane walked into the room. John walked in too. It was late in the day, and everyone was walking home after a long day at work. Jane said hi to ___”
The answer is John. This requires remembering the one occurrence of John about 25 words back.
Another nice example is from [4].
I grew up in France ... I speak fluent French.
Predicting French involves remembering France which may have occurred many sentences back.
Neural language models address this “long-range influence” problem. Additionally, they also address the short-range influence problem better, specifically with automatically learnable feature vectors. (Statistical approaches can also use features; however, these are human-engineered.)
Early Neural Language Model
One of the early neural language models, as described in [3], is feedforward in nature. Whereas the more recent approaches are now all recurrent, we start with this model because it has some insightful characteristics that will set the stage for the ones that follow.
As with many statistical language models, this model is kth-order Markovian. That is, for n greater than k, it assumes that P(wn | w1, w2, ..., wn-1) equals P(wn | wn-k, ..., wn-1).
It uses distributed representations for input words. Distributed representations, more recently also called word embeddings, go back to the 1980s. A distributed representation of a word maps it onto a d-dimensional numeric space. Distributed representations are learned by algorithms in ways that encourage semantically-related words to be near each other in this space.
The intuition for using learned distributed representations is the hope that they pick up useful features of words involving semantic relateness. For example, if one sees Paris and France in the same text, this might help the model to make sensible inferences involving the context French, such as in our language example: I speak fluent ___.
This model then concatenates the distributed representations of wn-k, ..., wn-1. This forms the input vector, of dimensionality k*d. Since we want the output to be a probability distribution over the entire lexicon of words, this network uses N output neurons, where N is the size of the lexicon.
A hidden layer sits between the input and output layers. As in any other multilayer neural network, the hidden layer maps the input to a vector of features. (The neurons are sigmoid-like, specifically tanh.) This mapping is done via learnable weights from each input neuron to each hidden neuron.
The mapping from the hidden layer to the output layer uses the softmax function. This is because we want the output to be a probability distribution over the output neurons. The softmax function is parametrized by weights, one per (hidden neuron, output neuron) pair.
The parameters of this network are learned in the usual way, i.e. via backpropagation using stochastic gradient descent. The distributed representations of the various words can also be learned during this process. Alternatively, such distributed representations can be pre-learned in some other way (e.g. word2vec) and just used here.
It is worth noting that when the size of the lexicon N is huge (say in the millions) the computations involving the output layer become the bottlenecks. This is because the predicted output on a particular input needs to be a probability distribution over the entire lexicon.
A more serious limitation is that this network, just like typical statistical models, cannot model influences among words that are more than k units apart. In our John-Jane example, we would need k to be at least 25 for us to have any hope of making the right inference.
Statistical models that explicitly compute P(wn | wn-k, ..., wn-1) require memory that grows exponentially with k. This neural network model scales better. It uses k*d*h + h*N parameters, where h is the number of hidden neurons. So the number of parameters increases only linearly with k.
Recurrent Neural Networks
Recurrent neural networks are appealing for language modeling because they are not constrained by k. That is, in principle, they can learn arbitrarily long-range influences. That said, it does require some special engineering for them to do so.
We’ll start by describing a basic RNN. We will see that it has difficulty learning long-range influences. We will then discuss modifications to this RNN to alleviate this issue.
Basic RNN
As a black box, i.e. from the input-output interface, an RNN looks like a feedforward neural network at first glance. It inputs a vector x and outputs a vector y.
There is a twist though. The RNN expects to receive, as input, a sequence of vectors x(1), ..., x(T), one vector at a time, and outputs a sequence of vectors y(1), ..., y(T).
The output y(t) at time t is influenced not only by x(t), rather by the entire sequence x(1), ..., x(t) of inputs received till then. This is what makes the RNN powerful. And the associated learning problem challenging. The RNN must carry state. Somehow it must summarize the history x(1), ..., x(t-1) into a state, which we will call h(t-1), from which it can predict y(t) well. When the input x(t) is presented, it first updates its state to h(t). In effect, x(t) becomes part of the history of the inputs it has seen. It then produces the output vector y(t) from h(t).
We can write this formally as
h(t) = f(h(t-1),x(t))y(t) = g(h(t))
In summary, the RNN may be viewed as having two parts:
A part that updates the state based on the current state and the new input.A part that transforms the current state to an output.
A part that updates the state based on the current state and the new input.
A part that transforms the current state to an output.
Part 1 is reminiscent of finite-state automata in computer science. Part 1+Part 2 is a finite-state transducer in computer science terms.
The state in an FSA may be viewed as a categorical variable. By contrast, the state in an RNN is a multi-dimensional vector. The latter facilitates compact distributed representations of states that can generalize well, a major advantage of modern neural representations.
The icing on the cake (to put it mildly) is that both the state-change and the state-to-output behaviors of the RNN can be learned automatically from a training set of (input sequence, output sequence) pairs.
The State-change and Output Functions
Following the usual neural network ‘design pattern’ inspired in part by biological neural networks and approved by statistics, we’ll use
f(h,x) = S1(Whh*h + Wxh*x)g(h)   = S2(Why*h)
Here S1 and S2 are often sigmoid-shaped functions, typically the tanh or the logistic function.
f(h,x) computes the next state as a linear combination of the current state and the next input pushed through the squashing function S1. g(h) computes the output as a linear transformation of the new state pushed through the squashing function S2.
RNNs in Language Modeling
A fundamental use case of RNNs is in language modeling. Recall that in language modeling, our primary interest is in estimating P(wn | w1, w2, ..., wn-1).
A natural way to model this in an RNN is to model wn-1 as the input vector x(n-1), the history w1, w2, ..., wn-2 as the state vector h(n-1) and the output y(n-1) as a probability vector over the lexicon of words. For these purposes, it is natural to use the softmax function as the output function S2. Rather than a squashing function. S2(u) exponentiates each component of u and then normalizes the resulting vector to form a probability distribution. That is, it divides each exponentiated value by the sum of the exponentiated values.
Structurally, this is analogous to the feedforward neural language model described earlier. The output dimensionality is N, the size of the lexicon. The input dimensionality is based on how words are represented. If words are represented locally, i.e. as a one-hot encoding, the input dimensionality is N. If words are represented in a distributed fashion, the input dimensionality is the dimensionality of this distributed representation. The state vector’s dimensionality is analogous to a hidden layer’s dimensionality, and controls the capacity of the corresponding feature space.
A sequence of words w1, w2, ..., wn-1, wn may be turned into a training instance for this RNN-based language model. The sequence of inputs is x(1), ..., x(n-1), the sequence of vectors of the first n-1 words. The target output y(n-1) associated with this sequence is derived from the word wn. It’s the probability vector with all its mass concentrated on the dimension that corresponds to this word.
Let’s pause to reflect on this. The output dimensionality is always N, as we are trying to learn a language model. This does have learning consequences as N is often very large. The input dimensionality may be distributed and compact or local and typically huge, i.e. N. There are trade-offs. A local representation imparts the model with a high capacity and is capable of making fine-grained distinctions. Each word has its own vector of weights to each hidden neuron. Models using distributed representations of words in the input layer are exponentially-more compact, may learn much faster, and potentially generalize better. They can leverage distributed representations that have already been learned by effective algorithms such as word2vec and enhancements, possibly from a huge corpus of documents.
Learning
The learnable parameters are the matrices of weights Whh, Wxh, and Why inside the state transition and the output functions. Whh controls the influence of the current state vector on the next state vector, Wxh the influence of the next input vector on the next state vector, and Why the influence of the next state vector on the next output vector.
To understand how learning happens, it helps to unfold the recurrent neural network in time, as depicted below.
Now consider the input x(t) at time t along with its target output y(t). Keeping in mind that the mapping we seek to learn is really to output close to y(t) on the sequence of inputs x(1), x(2), ..., x(t) we can depict the situation as a multilayer feedforward neural network unfolded in time as depicted below.
So we can use backpropagation to learn the weights inside the functions f and g. With a twist. All the f’s in the picture above must have the same weights. They are all the same f!
Okay, we can make this happen by summing up the changes to each weight in f across all the layers.
One way to think of this is as follows. Let’s imagine we have two versions of the network: the constrained version (whose weights are shared as explained in the previous paragraph) and the unconstrained version which is a multi-layer neural network with no weight sharing, i.e., each instance of f has its own weights. Before the learning on the pair (x(t), y(t)), we’ll initialize each weight in the unconstrained version to the corresponding weight from the constrained version. We then run one iteration of online backpropagation (aka stochastic gradient descent) to update the weights on the unconstrained version. From these, we derive the new weights on the constrained version by taking, for each constrained weight, the average of the unconstrained weights that map to it.
Well, all this sounds great in principle. However, note that the unfolded network can be very deep! So if there is a long-range influence, meaning that x(t-k) strongly influences y(t) for a large k (say k = 100), the back-propagated error signal may get very weak. This is because every instance in which the error is back-propagated through f attenuates the error since f is a squashing function.
Let’s expand on the last sentence in the previous paragraph, in the setting below.
To simplify the explanation, we use a feedforward neural network with two hidden layers. Plus the input and the output layer. All layers have a single neuron each. All layers except the input one have a sigmoidal transfer function. hi denotes the output of the neuron at layer i, and ni denotes the input to this neuron. So we have hi = s(ni) where s denotes the sigmoid function. We also have ni = w(i-1)*h(i-1).
Note that to simplify the notation h0 denotes the input and h3 the output.
In this illustration, we will use the loss function L = 1⁄2 (y-h3)2. The backpropagation algorithm computes the gradient of L with respect to each weight wi (applying the chain rule when needed) to decide on the change to make to the weight. More precisely,
Delta wi = -n*dL/dwi
Here n is the learning rate. The ‘-’ is there because we are doing gradient descent (since we want to minimize the error).
So now it comes down to computing dL/dwi for the various weights wi.
Let’s start with w2.
By the chain rule,
dL/dw2 = (dL/dh3)*(dh3/dn3)*(dn3/dw2) = 2(y-h3)*s(n3)*(1-s(n3))*h2
Admittedly we are overloading notation a bit. In the above, h2, h3 and n3 denote both the variables (neurons) and their particular values on the input x = h0. We are also using the fact that the derivative of the sigmoid function ds/dn equals s(n)*(1-s(n)). Also that, for i > 0, dni/dw(i-1) equals h(i-1).
Similarly, with a slightly deeper application of the chain rule, we get
dL/dw1 = (dL/dh3)*(dh3/dn3)*(dn3/dh2)*(dh2/dn2)*(dn2/dw1) =
2(y-h3)*s(n3)*(1-s(n3))*w2*s(n2)*(1-s(n2)*h1
Now imagine adding more hidden layers to our network (keeping the topology and the transfer functions the same) and computing dL/dw0.
dL/dw0 = 2(y-h(K))*s(nK)*(1-s(nK))*w(K-1)*s(n(K-1))*(1-s(n(K-1))*...*w1*s(n1)*(1-s(n1))*h0
Consider any one term s(ni)*(1-s(ni)). This product is always less than 1⁄4. Combining all such pairs in dL/dw0 produces a multiplicative factor whose value is less than (1⁄4)^K. In view of this, as K increases, dL/dw0 approaches 0 very very quickly. This is the vanishing gradient problem. This has a potentially disastrous consequence in our example. We may never learn w0.
A Long-range Example
Next, let’s see a realistic example with long-range influences. We will use it to illustrate that learning such influences is the issue with basic RNNs, not the ability to represent them. This same example will also help illustrate the mechanisms of a more advanced RNN that alleviate this issue.
We’d like to process a long sequence of English words and output a 1 if the word terrorist appears at least once, and 0 if not. Were terrorist to occur only once and very early on in the sequence, this event needs to be remembered until the very end.
Why not just output a 1 immediately after terrorist is detected, else keep going? That would be tweaking the mechanisms of a general solution to this particular problem. This tweak would not generalize to the broader class of problems exhibiting long-range influences, of which this is just one illustrative example.
The XOR example was used in the feedforward neural networks setting in exactly the same way. Viewed as a problem to solve, XOR is trivial. Just input two binary numbers and output the binary number that is their exclusive OR. However, viewed as an example to illustrate issues with certain feedforward architectures and to test improvements, it has turned out to be quite useful. The XOR is linearly inseparable. This is why classifiers capable of only linear discrimination fail. Indeed in the 1980s, the XOR was often used as a benchmark to test and demonstrate the learning ability of backpropagation in the setting of multi-layer perceptrons — feedforward neural networks with a hidden layer and sigmoidal neurons.
Okay, back to our example. Let’s refine the problem description for our purposes here. The input x(1), ..., x(t) is a sequence of vectors where x(i) represents the ith word. The output y(t) is 1 if the word terrorist appears in this sequence and 0 if not. So the output is a binary sequence aligned with the input sequence and has a very simple structure: 0*1*.
We will represent a word as a one-hot encoded vector. The vector’s dimensionality is the size of the lexicon. The bit corresponding to the word being represented is 1, the rest are 0.
Now consider a basic RNN for this problem. Its input and output structure has already been specified. What about the state? Let’s set the state vector’s dimensionality to be the same as that of the input vector. We’d like hi(t) to be close to 1 if the word indexed by i was seen at least once among x(1), ..., x(t), and close to 0 if not.
This is certainly feasible representationally, with the choice
hi(t) = sigmoid(a*hi(t-1) + b*xi(t) + c)
It is easy to choose a, b, and c, for example, a = b = 1 and c slightly negative, so that hi(t) is essentially an OR of hi(t-1) and xi(t). This is what we want.
Once we have processed the entire sequence, we can simply set y(t) to hi(t), where i indexes the particular word we care about (in our case, terrorist). We can write this in the form y = g(W*h(t)) where g is a sigmoid. W is the vector whose ith position is sufficiently positive (e.g. 1) and the rest are 0.
Now let’s look at this situation from the perspective of learning. Imagine that we have a training set of (document, label) pairs where label is 1 if a certain word appears somewhere in the document and 0 if not. We can assume that the labeling is consistent, i.e. this “certain word” is always the same. However, we are not told what that word is.
So there are two issues here: (i) the learner must deduce which of the potentially millions of words best discriminates the two classes and (ii) learn only sequentially as described earlier in the basic RNN section.
Focusing on (ii), backpropagation-through-time may need to propagate error gradients far back in time in many cases. Such gradients can vanish as discussed in an earlier section. The consequence of this is that if in most positive instances, the last occurrence of the “certain word” (in our example, terrorist) appears well before the last word in the document, the learner will not be able to establish the connection between this occurrence and the output is 1. If our positive instances are at least somewhat long documents, in most of them, the last occurrence of the word terrorist will indeed be well before the document’s end.
Before closing this section, we’d like to add that now that we have opened the door to the labeling driving the learning, this problem becomes broader in its import. We can pose any binary classification problem on text this way, so long as we have a rich enough data set. It’s up to the recurrent neural network solution to figure out which words and which sequential structure helps to solve the problem.
That said if one’s aim is to solve such a problem and if there isn’t a compelling case to be made for sequential learning, one should try non-recurrent binary classifier algorithms first. Such as feedforward neural networks, random forests, naive Bayes classifiers, ...
Gated Recurrent Unit (GRU)
We now turn our attention to more advanced recurrent neural networks, capable of capturing long-range influences. Two such networks in wide use are the LSTM and the GRU.
We will describe the GRU as its mechanisms are simpler while remaining effective for capturing long-range influences.
At a high level, the GRU works in the same fashion as a basic RNN. The input is a sequence of vectors presented one by one. The GRU maintains state, capturing some key aspect of what it has seen so far. This state combined with the next input determines the next state.
The advancement comes in the state transition function.
We’ll use the example we just introduced to build up the description of the GRU.
Consider the state update function of the basic RNN for this example
h(t) = sigmoid(A*h(t-1) + B*x(t) + C)
The GRU will build up h(t) from h(t-1) and x(t) in a different way. It introduces the notion of a new memory hnew(t) derived from h(t-1) and x(t). The process of deriving this new memory from h(t-1) and x(t) has a mechanism that first runs each component of h(t-1) through a filtering gate. Think of this as letting the network selectively pay differing attention to different components of h(t-1) for this purpose.
This filter, let’s call it si(t), takes on a value between 0 and 1. Filtering involves multiplying hi(t-1) by si(t). In vector notation we write this as s(t) o h(t-1).
The filtered version of h(t-1) is then linearly combined with x(t) and passed through a component-wise tanh. The full equation is
hnew(t) = tanh(A*x(t) + B*(s(t) o h(t-1)) + C)
The new memory is then combined with the old memory to form the updated memory, in a manner that will be described soon. By contrast, in the basic RNN, the updated memory is formed only from the old memory.
What would be a good hnew(t) in our example? Let’s capture the desired behavior in pseudocode.
IF i indexes terrorist and xi(t) is 1    hnew,i(t) = ~ 1IF i indexes terrorist and hi(t-1) is ~ 1    hnew,i(t) = ~ 1IF i indexes some other word    hnew,i(t) = ~ 0ALL ELSE    hnew,i(t) = ~ 0
hnew,i(t) is close to 0 for any word i other than terrorist. hnew,terrorist(t) is close to 1 if the new input x(t) is terrorist or if terrorist has been seen previously, in which case hi(t-1) would be close to 1.
Can we choose the learning parameters to emulate this desired behavior?
First, we will set C to the vector of 0s.
The THEN clause of the first IF can be executed via an appropriate choice of matrix A. Specifically, set Aii equal to 1 and all the rest of A’s elements to 0.
The THEN clause of the second IF can be executed by choosing si(t) to be near 1 for i indexing terrorist and near 0 for the rest. Choose B as a matrix of positive values, e.g. all 1s.
Under these conditions on A, B, and s(t), hnew,i(t) will be set near 0 when neither of the first two THEN clauses triggers.
At this stage the reader might wonder, what have we really gained? If the last occurrence of the word terrorist appeared way before the current t, will not computing hnew,i(t) from the long chain of triggerings of the second THEN clause — the one whose antecedent is “i indexes terrorist and hi(t-1) is near 1” — have the same diluting effect.
The reader is right to wonder this way. We have not yet completed the GRU’s description. Once we have all the mechanisms in place we will get a better sense of how they all come together to alleviate the vanishing gradient problem.
The GRU uses a second gate, called the update gate, to control the relative contributions of the old and the new memories towards the updated memory. By contrast, the basic RNN derives the updated memory just from the old memory.
Before diving into the update gate, let’s complete the description of the filtering gate.
Filtering Gate Equation
This is
s(t) = sigmoid(D*x(t) + E*h(t-1) + F)
Here D, E, and F are the matrices and vectors of the learnable parameters that control the behavior of the gate.
In our terrorist example, the behavior we sought from s(t) was to simply project out the value of hi(t-1), where i indexes the word terrorist. How well we can learn this behavior automatically from a training set is a whole different story.
Example to illustrate the update gate mechanism
To build intuition about the update gate and the role it plays, a different example will help.
We’d like to do anomaly detection in a time series using an RNN. We want the anomaly detector to not only detect anomalies but also adapt to new normals. Let’s illustrate this with an example.
1, 2, 3, 2, 4, 2, 3, 2, 3, 23, 22, 25, 24, 22, 23, 24, 25, 21, ...
We process this time series from left to right. The value 23 is way outside the range of the values we have seen to this point, so we deem it anomalous. As we should. As we continue to process values to the right of this anomaly, we see that a new normal is developing. (Specifically, a level shift.) At some point soon the anomaly detector should adapt to this, and start treating it as the ‘normal’. For example, stop deeming values between 20 and 25 as anomalies.
We’d also like to be able to control ‘at some point soon’ more finely by providing a labeled data set. By this, we mean that a time series comes labeled with which points are anomalous and which not. The RNN should be able to deduce, for any particular time series, how aggressively or conservatively we want to transition to the ‘new normal’. This would be the time-series specific behavior inferred from the historical labeling.
One way to approach this is to train the RNN to forecast the next value in the time series. If the actual next value differs a lot from the forecasted next value then deem that value as being anomalous.
RNNs are in principle suited to one-step forecasting because this problem is similar to learning a language model. In both cases, we want to predict the next input from the historical inputs.
The model that the RNN learns during its incremental learning on the target of forecasting the next value may be interpreted as the current model of ‘normal values’.
Now suppose that this RNN is a basic RNN. What happens when it encounters the first occurrence of 23. We hope it flags it as anomalous. What happens next? It incrementally trains on 23, as it would on any new value. The effect of this is that, as we see more values at this level, the model will gradually adjust to this new level. That is, it is building a ‘new normal’. So far so good. It is going in the direction we want.
If this 23 was an outlier, i.e. a spike, not a level shift, the RNN would soon adjust back to the old normal. So all seems good. Let’s look at this more closely though. As the basic RNN is incapable of remembering the old normal (the one based on the values in the time series to the left of the first occurrence of 23) it must unlearn its accommodation of 23 towards an evolving new normal. Which in hindsight it can see was an anomaly.
The GRU addresses this situation better. Just like the basic RNN, the GRU learns to adapt its ‘normal’ by forecasting the next value and adjusting its model based on the forecast error. Unlike the basic RNN, the GRU works with two models: one a slower evolving model, captured in h(t), the second a faster-evolving model, captured in hnew(t). In effect, the GRU can remember both the ‘old normal’ and a tentative ‘new normal’ at the same time. Its composite model is a combination of these two. This gives it the flexibility that should the ‘new normal’ later turn out to be a ‘false alarm’, i.e. an anomaly, it can just use the ‘old normal’. By contrast, the basic RNN would have to unlearn it. It also lets the GRU adapt quicker to the new normal in case it is not a ‘false alarm’.
This behavior is summarized in the pseudocode below.
Evolve two models, a slow one and a fast one, based on forecasting and feedback from forecasting errors.IF the new fast model is somehow deemed as capturing a transient phenomenon   continue treating the 'old model' as the normalELSE   evolve towards the 'new fast model' as the normal
Update Gate Usage Equation
Let’s call this gate u. ui(t) is between 0 and 1. The equation for the updated memory using this gate is
hi(t) = (1-ui(t))*hi(t-1) + ui(t)*hnew,i(t)
which, in vector notation, is
h(t) = (1-u(t)) o h(t-1) + u(t) o hnew(t)
Anomaly Detection Example Refined
Next, let’s refine the discussion on our anomaly detection example. We’ll use one-dimensional input and output vectors since we are working with a time series. For the state vector, we might consider a higher dimensionality. Whatever we feel is appropriate to capture the ‘normal’ characteristics of the recent values of the time series. For instance, if we wish to capture not only (a smoothed version) of the most recent value of the time series but also its trend, we might choose two dimensions.
By choosing the dimensions of h and hnew with care, we can detect different types of anomalies. To detect point anomalies, two-dimensional state vectors may suffice. One dimension models the central tendency, e.g. mean or median, the second the dispersion, e.g. standard deviation or a more robust version. To, additionally, detect trend changes, we may want additional dimensions, ones that model characteristics of trends.
Like h(t), hnew(t) is also a model of recent values. That said, as hinted by the name, hnew(t) is a model of even more recent values than h(t). For example, h(t) might be a slow-moving average and hnew(t) a fast-moving average.
Now suppose the most recent values in the time series are anomalous. Let’s say we know this fact. That is, the anomalies are labeled. We can capture this in the value of the update gate u(t) where t is one of these anomalous time points. This gives us control over whether we want the most recent values (which we know are anomalous) to update our ‘normalcy’ model or not.
For instance, we might not want to alter our ‘normalcy’ model if we are in the midst of an anomaly. We can make this happen by setting the components of u(t) close to 0. If we are in a ‘normal’ region we want our normalcy model to adapt quicker to the data.
Update Gate Equation
We have seen how the update gate’s value is used in constructing the updated memory from the new memory and the old memory. But how is the update gate’s value itself calculated? This is done as below.
u(t) = sigmoid(A*x(t) + B*h(t-1) + C)
Here A, B, and C are the matrices and vectors of the learnable parameters.
Putting All the Equations Together
We have covered all the mechanisms inside the GRU. Let’s put all the equations together and review the end-to-end flow. At a single time t that is.
s(t)    = sigmoid(A*x(t) + B*h(t-1) + C)hnew(t) = tanh(D*x(t) + E*(s(t) o h(t-1)) + F)u(t)    = sigmoid(G*x(t) + H*h(t-1) + I)h(t)    = (1-u(t)) o h(t-1) + u(t) o hnew(t)
Let’s now walk through the end-to-end flow at a single time t. We assume that the learning has already happened.
The new input x(t) arrives at time t. The RNN’s current memory state is h(t-1). First, from these two we compute the value s(t) of the filtering gate. Now we have what we need to compute the value hnew(t) of the new memory. So we do that. Next, we compute the update gate’s value u(t) from x(t) and h(t-1). Note that the equations for computing s(t) and u(t) have the same form. The differing behaviors arise from the different learned parameters. Finally, we compute the updated memory h(t) from the old memory h(t-1) and the new memory hnew(t) using the update gate u(t) to combine the two.
“Learning”
The parameters to be learned are the matrices A, B, D, E, G, and H and the vectors C, F, and I.
We have placed this section’s name in quotes intentionally. Rather than describe learning in the usual purely supervised way, we will just convey intuition on how the various mechanisms in the GRU might be learned from some combination of human-driven biasing and self-supervised or supervised machine learning.
We will illustrate these in the anomaly detection example. In this illustration, we will be taking some liberties with the GRU. That is, we will tweak its mechanisms as appropriate. Our aim is to provide insights into the mechanisms and variations by way of an example.
Two for one: Anomaly detection also solves forecasting
We think it's helpful to add that buried within the anomaly detection approach we describe below are certain time series forecasting tasks. We want to call this out here because, in the process of describing how to use the (somewhat tweaked) GRU to solve time series anomaly detection problems, we will also be implicitly covering how to solve time series forecasting problems. So you get insights into solving both problems using RNNs.
First, anomalies not labeled
We are given a time series. If the anomalies are not labeled, we can train a GRU on the target of forecasting the next value in the time series. This process forces the GRU to learn a model of (recent) ‘normalcy’, adapting it along the way as ‘new normals’ develop.
h(t) should model the ‘old normal’ at time t. h could be multi-dimensional. For example, one component could describe the mean value in the normal region, a second the trend.
hnew(t) is similar to h(t) except that it is more heavily influenced by the recent values. For example, h(t) might behave as a slower moving average, hnew(t) as a faster one. Possibly with trend components so that we can discern if hnew(t)’s trend differs from h(t)’s.
How can we nudge the GRU into learning that hnew(t) should model more recent values than does h(t)? We could learn the parameters in hnew(t)’s equation by explicitly forcing hnew(t) to forecast a short-horizon forecasting target. Such as y(t) to equal x(t-1), i.e. a one-step forecast.
How can we make h(t) model a slower-adapting variant of hnew(t), i.e. model normalcy at a coarser time scale than does hnew(t)? (Except when we want h(t) to chase hnew(t) such as when a ‘new normal’ is developing.)
A sensible way to force h(t) to move slower is to have it forecast a longer-horizon target, i.e. set y(t) to equal x(t-k) for sufficiently large k. While this is an appealing idea, the needed architecture is two forecasters on the same input at two different horizons. This is not what the GRU is. The GRU must learn all its parameters from a single target, say y(t) equal to x(t-1). Plus this doesn’t accommodate the exception, i.e. sometimes we want h(t) to chase h(t-1). We’d need another mechanism for this.
To see what we have to work with in the GRU, let’s write out the state update equation again
h(t) = (1-u(t)) o h(t-1) + u(t) o hnew(t)
Assuming h(t-1) is slow-moving, and hnew(t) is very different from h(t-1), we’d want h(t) to chase h(t-1). We can make this happen by setting the components of u(t) to close to 0. But how?
First let’s write out the update gate equation again
u(t) = sigmoid(A*x(t) + B*h(t-1) + C)
What if we modified it so that hnew(t-1) also appears on the right-hand side?
This would in principle allow us to learn the behavior
IF hnew(t) is very different from h(t-1)    set u(t) close to 0else    set u(t) close to 1
This also potentially helps the model adapt quicker to a developing ‘new normal’. Let’s illustrate this with an example.
Consider a level shift, such as the one at the bolded value below.
t: 1  2  3  4  5  6  7  8x: 1  1  1  1  4  4  4  4
hnew(5) is 4. h(4) is 1. So we’d want h(5) to stay close to 1. We interpret this as “we don’t know yet whether 4 is a transient anomaly or the being of a new normal, so let’s have our old normal adapt towards the new value but slowly”.
As we continue moving to the right, h(t) will slowly start moving towards 4, i.e. towards hnew(t+1). This will in turn drive u(t) towards 1, which will further accelerate this movement, i.e. have a snowball effect.
We can summarize this in the following behavior we expect on this level shift. At the level shift event, initially, the old model moves slowly as it doesn’t yet know it's a level shift. Once the evidence that it is indeed a level shift accumulates the model accelerates its move towards the new normal.
On the other hand, if the shift was a transient anomaly (e.g. spike or dip), such as in the time series below
t: 1  2  3  4  5  6  7  8x: 1  1  1  1  4  1  1  1
h would stay close to the old normal throughout.
Feature Engineering
What if we don’t like the idea of modifying the update gate equation? We might be able to achieve a similar effect by feature engineering. E.g. by working off a transformed version of the time series such as x(t)-x(t-1). The intuition here is that this time series reveals where the original time series changes more explicitly. A possibly more robust variant of this would be to define our featurized time series as a suitable faster-moving average minus a suitable slower-moving average.
We can even imagine a sliding Kernel that explicitly distinguishes between transient changes and more stable ones. This Kernel function would have a high value if it is centered on an outlier deemed a transient event and low if not.
Armed with such a Kernel, should we use it to generate the features or the labels? The former would mean that we are replacing the input x(t) by a feature-vector f(t) in which one of the features is the value of this Kernel function centered on t. The latter would mean that we set y(t) based on the value of this Kernel function at time t.
We won’t dive into this question — just surface a few thoughts. It is appealing to use the Kernel function to generate the labels because they would then explicitly encode whether an event is transient or more persistent, exactly the right teaching signal for our anomaly-detecting GRU. On the other hand, we have to consider the risk that this labeling is biased. Generally speaking, label bias is more concerning than feature bias. Label bias manifests itself as “you are teaching the model the wrong thing”.
Anomalies are labeled: Human or computer
Now let’s delve into learning the GRU’s parameters when the anomalies are labeled by humans, computers, or a combination. In this section, by anomalies, we mean transient ones.
We can use the labels to set u(t).
IF x(t) is labeled ‘anomalous’  Set u(t) close to 0    // Direct GRU to continue 'old normal'ELSE   Set u(t) close to 1   // Direct GRU to move towards 'most recent                        // values'
Notice that we have taken direct control of the update gate here.
Let’s look into Direct GRU to move towards ‘most recent values’ more closely. If the new values in the time series continue the old normal, then the update will also continue the old normal. If the new values are creating a different new normal, as in our level shift example, this will have the effect of the GRU adapting quickly to it.
Okay, so this takes care of when to update the ‘normal’ and when not to. But what about how to update the two models— recent and more-recent — themselves?
For the how it is tempting to rearrange the various mechanisms of the GRU for our use case. Sure, we lose the ability to use the GRU as a black-box. On the other hand, we learn what we can do with other combinations of these mechanisms.
We will continue to use h(t), hnew(t) and u(t). h(t) and hnew(t) will each learn to forecast x(t) albeit at different horizons. So each will be equipped with its own target y(t) = x(t+k1) for h(t) and ynew(t) = x(t+k2) for hnew(t).
u(t) will be set from the anomaly labels in the manner described earlier.
This model may be viewed as blending self-supervised forecasting with supervised anomaly detection.
What about the filter gate?
What about the mechanism s(t) in the GRU? Can it be profitably used here? First, let’s remind ourselves what it actually does. It projects out those components of h(t-1), in the context of x(t), that matter to the task at hand. Which in our case is to detect anomalies.
Well, actually under our hood we have two instances of another task: forecasting. As we have already used the update gate in the anomaly detection task (to switch between forecasting models), let’s try to use it inside a forecasting task itself.
Forecasting and Filter Gate
We seek a good forecast of x(t) at a certain horizon. Such a model maintains a state that describes the current ‘normalcy’. This is represented in a possibly multi-dimensional vector. Such as a two-dimensional vector where one dimension captures a smoothed estimate of the most recent value; the second the local estimated trend at that value.
Here is a physical ‘motion’ analogy. Say we’d like to predict the future location of a point moving in a certain Euclidean space. Statistical estimates of the current location and the current velocity are helpful in this regard.
In this setting, we can speculate how the filtering gate might come into play. Depending on the time series some features of the ‘normalcy’ state may perform better than others on our forecasting task. For a short horizon such as one-step, and for a time series that doesn’t have a nice linear trend (although it may have seasonalities), an estimate of the most recent value may be the best forecaster. For a time series with a durable linear trend and for a longer horizon, the local trend dimension of the ‘normalcy’ state may be very predictive.
The reasoning of the previous paragraph starts getting stronger as we add more dimensions to the ‘normalcy’ state model. Such as for various seasonalities, or lags.
Using s(t) can in principle select out those components of the normalcy state that matter to the task at hand. This plays a role similar to that of feature selection or pruning in predictive models. Such pruning often helps. This is known both theoretically and empirically. It forces the algorithm to learn models with a bias towards simplicity. Simple models generally generalize better. Occam’s razor.
But how do we learn s(t)?
Okay, so we think using s(t) can help. Let’s dig a bit deeper to see how we might learn what we hope s(t) can learn.
We will illustrate this in a basic RNN for time series forecasting.
From the input x(t) we extract certain features such as its value, i.e. x(t) itself, and lags x(t)-x(t-k) for k=1, 2, .... (In order to compute the lags we need to remember previous values of the time series.) Let v(t) denote x(t)’s feature vector. The state vector h(t) mirrors this feature vector. For example, h1(t) may model a smoothed version of x(t), h2(t) a smoothed version of x(t)-x(t-1), and so on.
The state vector models the characteristics of values of the time series that help forecast its future values. Such as an estimate of the most recent value, an estimate of the most recent trend, and possibly the various seasonalities.
The RNN may now be expressed as
h(t) = f(h(t-1),v(t))y(t) = g(h(t))
We can set y(t)=x(t+h) as the target. That is, we seek an h-step forecast of the time series x.
f specifies how to derive the new state vector h(t) from the combination of the previous state vector h(t-1) and the feature vector v(t) of the new value x(t).
g specifies how to derive the forecast from the current state vector h(t). For example, if h(t) just has two features: a smoothed estimate of the current value and a smoothed estimate of the trend at it, we might just multiply the second with a parameter learned from the target (this is especially useful to get the influence from different horizons, such as h being small versus large) and add the first one to it. In equation form, this would be
y(t) = h1(t) + b*h2(t)
The second term on the right-hand side models a trend-based correction to the first term.
Learning happens by propagating back the forecast error and adjusting the parameters implicit in g and f.
So how should we inject s(t) into the mix? First, let’s drastically simplify. Let’s consider
h(t) = a*h(t-1) + (1-a)*v(t)
This is not so naive for forecasting. In fact, this is how forecasters based on exponential smoothing work. Here a is the only learnable parameter in f.
Note that this approach also matches the role we had for h(t), to serve as a smoothed version of v(t).
Now we are ready to inject s(t).
h(t) = s(t) O h(t-1) + (1-s(t))o v(t)
That is,
hi(t) = si(t)*hi(t-1) + (1-si(t))*vi(t)
This update rule is potentially richer as we now have component-level control over the state vector update. Effectively we have gained the ability to do soft feature selection during this process.
GRU and The Vanishing Gradient Problem
What about the vanishing gradient problem? Is the GRU more robust against it? We won’t answer this question definitively; rather we will give some tantalizing hints on what might be at play.
Let’s first write out the state update equations of the GRU vs the basic RNN.
h(t) = (1-u(t)) o h(t-1) + u(t) o hnew(t)  (GRU)h(t) = sigmoid(A*h(t-1) + B*x(t) + C)      (Basic RNN)
What jumps out? The sigmoid in the second equation. From our earlier discussion on the vanishing gradient problem, it seems that sigmoid-like transfer functions are implicated. Squashing functions have a near-zero slope in their tails. This can aid in the error gradients vanishing rather quickly, as backpropagation through time back-propagates the error seen at the output multiple time segments in the reverse direction through the chain of these sigmoids.
By contrast, the GRU’s state update equation does not use a squashing nonlinearity.
Stacked RNNs
Now that we understand the GRU and the basic RNN more deeply, let’s see a significant generalization. By
Consider the state update function of an RNN
h(t) = f(h(t-1),x(t))
In the ones we have seen so far (the basic RNN and the GRU),
f(h,x) = S1(Whh*h + Wxh*x)
We may read this as “h(t) is obtained from h(t-1) plus x(t) in a feedforward architecture with zero hidden layers”.
A stacked RNN simply adds one or more hidden layers to this computation. That is, the state update function becomes ‘deep’. Not surprisingly, this makes the RNN architecture richer, often performing better on certain tasks.
A stacked RNN unfolded in time looks like this
Now, in addition to being deep in time, we are also deep in the state update function. We now have L functions f1, ..., fL with their own learnable parameters. We can write this as
f(h(l),h(l-1),l) = S1(Whh(l)*h(l) + Wxh(l)*h(l-1)), l = 1 to L
where h(0) denotes x.
The state representation at time t itself is richer. h(t) = (h(0, t), h(1, t),..., h(L, t)). This may be viewed as a hierarchical representation of state in order of increasing coarseness. As in deep forward neural networks.
Inference
Next, we talk inference. That is, how we might use a trained RNN to answer various questions.
We can run a sequence of words w1, w2, ..., wn-1 through a trained RNN and produce a probability distribution over the likely next word. Typically, we want more than this. We want the most likely extension of a certain length k of the given sequence. That is, we want
wn, wn+1, ..., wn+k-1` = argmax wn, wn+1, ..., wn+k-1` P(wn, wn+1, ..., wn+k-1 | w1, w2, ..., wn-1)
This inference problem is called the decoding problem. When k is 1, the solution is just the word with the highest probability in the RNN’s output vector after the RNN has processed the words w1, w2, ..., wn-1 sequentially as input. For the general case of k, the decoding problem is intractable. That is, we cannot hope to find the most likely extension efficiently. So we resort to heuristic methods.
Greedy decoding
One simple approach to finding a good extension of length k is to start by finding the highest-probability word in the output vector, inputting this word to the RNN (so that it outputs the probability distribution over the next word), finding the highest-probability word again, and repeating the process. This approach is called greedy because it makes the best local decision in every iteration.
This approach can work well when, in every greedy step, adding the highest-probability word in the output vector moves us towards a globally optimal solution. As in the example below.
Example: Consider a language model trained on national, state, and county park names. Consider x(1) = yellowstone. The greedy extension will be x^(2) = national.
Not surprisingly, such a happy circumstance is not always the case. As in the example below.
Example: Consider a language model trained from a multiset of organization names. Examples of organization names are Bank of America, Google, National Institutes of Health, ... The term multiset means that organization names may be repeated in this list. Now consider the problem of finding the most popular organization name in this list. The first step of the greedy method would pick bank or the or whichever word occurs most frequently in this list. The most popular organization name (say Google) may not start with this first word.
Beam search
As mentioned earlier, optimal decoding is in general intractable (NP-hard). There are many heuristic approaches that try to do better than greedy decoding, at varying costs of running time and memory requirements. Beam search has turned out to be an especially attractive one in sequence-to-sequence labeling use cases (especially language translation). So that is what we will describe here.
Beam search keeps the k best prefix sequences. In every iteration it considers all possible 1-step extensions of these k prefixes, computes their probabilities, and forms a new list of the k best prefix sequences thus far. It keeps going until the goal is reached. In our scenario this would be when the extended sequence lengths are T. It then pulls out the highest-scoring sequence in its list of k candidates as the final answer.
References
Statistical Language Models. From simple to ++, with use cases... | by Arun JagotaCS 224D: Deep Learning for NLPNeural net language modelsUnderstanding LSTM Networks — colah’s blogExponential Smoothing Approaches In Time Series Forecasting | by Arun Jagota
Statistical Language Models. From simple to ++, with use cases... | by Arun Jagota
CS 224D: Deep Learning for NLP
Neural net language models
Understanding LSTM Networks — colah’s blog
Exponential Smoothing Approaches In Time Series Forecasting | by Arun Jagota