Word Sequence Decoding in Seq2Seq Architectures
By: Prakhar Mishra
Tags: Machine Learning, Deep Learning, Neural Networks, Naturallanguageprocessing, Naturallanguagegeneration

Natural Language Generation (NLG) is a task of generating text. Natural Language Generation tasks like Machine Translation, Summarization, Dialogue Systems have a core component of generating the sequence of words as an output conditioned on a given input. For example — For a machine translation system, given an input sentence in English, the model needs to generate its French translation. Today most such systems are built on Encoder-Decoder architecture and it’s variations. Fig.1 shows a diagrammatic glimpse of such an architecture.

Fig.1 from Source

In the above-shown image, Encoder is responsible for capturing the full context in the source/input language, whereas, Decoder is responsible to use this information and output its translation in the desired language. Such models are trained with a huge parallel corpus of sentences for both the languages. Getting into the nitty-gritty of this architecture is beyond the scope of this blog. Please read more here. Also, actual Google’s NMT system has an attention component to it as well. To read about attention networks please read this.

It is quite common to output a probability distribution using Softmax activation function across vocabulary at each decoder time-step. Choosing the final output sequence after the model has been trained depends on the decoding strategy that one uses.

Here, we will discuss 3 decoding strategies that are widely used in practice during inference time—

1. Greedy Search

This strategy selects the most probable word (i.e. argmax) from the model’s vocabulary at each decoding time-step as the candidate to output sequence.

Decoder Segment

The problem with this approach is that once the output is chosen at any time-step t, we don’t get the flexibility to go back and change our choice. It is seen in practice that greedy decoding strategy is prone to have grammatical errors in the generated text. It will result in choosing the best at any time-step t but that might not necessarily give the best when considering the full sentence to be grammatically correct and sensible.

2. Beam Search with Fixed Beam Size

The beam search strategy tries to find an output sequence having a maximum likelihood. It does this by extending the greedy sampling to Top-k sampling strategy. At any time-step t, it considers top-k most probable words as the candidate words for that step. Here, k is called the beam size. Mathematically, we are trying to maximize the below mentioned equation —

Below diagram shows how it happens in practice for k=3 —

Beam Decoding with k=3 at each time-step

Here, we search for a high scoring output sequence by keeping track of top k vocabulary outputs at each time-step while decoding. We usually stop our search when end-of-sentence token (<eos>) is reached or till time-step t for all or at-least some n output sequences. We also normalize by length, to avoid any biasness induced while scoring any sequence.

Normalized Beam Search Scoring

Here, beam size acts as a trade-off variable between time complexity and accuracy. Let’s analyze the minimum and maximum values that k can take:

When (k=1) — It behaves like Greedy search where argmax at any time-step t is fed to later consecutive step.

— It behaves like Greedy search where argmax at any time-step is fed to later consecutive step. When (k=Size of Vocabulary) — It behaves like an Exhaustive search where possible words at each time-step are whole of vocabulary and of which each gives probability distribution over the next set of vocabulary for later consecutive step.

Time Complexity Analysis

3. Beam Search with Variable Beam Size [1]

The notion of variable beam size was developed considering the limitations of fixed-beam size. Fixed beam size might not necessarily be an optimal choice at each decoding time-step.

Let’s understand it with an example — Let the probability values for the top 5 words at any time-step t are comparable. In such cases, fixing beam size(k) we might miss out on relevant candidates resulting in information loss. Whereas, let top 2 words at any time-step t are comparable and rest are way low. In such cases, fixing beam size(k) we might add noise by adding not so relevant candidates.

Instead, the beam size should be a function of the probability distribution at any decoding time-step (t). In [1] the author talks about various techniques that can be used to derive this relation. Also, if you notice carefully then this can also be seen as a binning problem that can be discretizatized based on entropy measures. Please read more here.

4. Temperature Induced Softmax

This isn’t a decoding strategy but can be used in-hand with any of the above-mentioned searches. At any decoding step, we typically use the Softmax activation function to deliver probability distribution over our vocabulary. Instead of using plain Softmax we use the modified version of it, shown below —

Temperature Induced Softmax

Here, T is the temperature variable. It can be easily seen that higher values of T, would result in fattened distribution (giving almost same probabilities to all), whereas, lower values of T would result in peaked distribution (giving high probabilities to some).

Although the blog is majorly focused on talking about word sequence decoding, it goes unsaid that these inference time decoding sequence algorithms can be used for any sequence decoding tasks.

Implementation of the above mentioned approaches can be found here.

Check out my blog repository at https://prakhartechviz.blogspot.com

Further Readings

Feel free to share your thoughts :)