Question: What is one of the most severe problems of NMT?
Evidence: An NMT system consists of an encoder which automatically learns the characteristics of a source sentence into fix-length context vectors and a decoder that recursively combines the produced context vectors with the previous target word to generate the most probable word from a target vocabulary.
More specifically, a bidirectional recurrent encoder reads every words INLINEFORM0 of a source sentence INLINEFORM1 and encodes a representation INLINEFORM2 of the sentence into a fixed-length vector INLINEFORM3 concatinated from those of the forward and backward directions: INLINEFORM4 
Here INLINEFORM0 is the one-hot vector of the word INLINEFORM1 and INLINEFORM2 is the word embedding matrix which is shared across the source words. INLINEFORM3 is the recurrent unit computing the current hidden state of the encoder based on the previous hidden state. INLINEFORM4 is then called an annotation vector, which encodes the source sentence up to the time INLINEFORM5 from both forward and backward directions. Recurrent units in NMT can be a simple recurrent neural network unit (RNN), a Long Short-Term Memory unit (LSTM) BIBREF3 or a Gated Recurrent Unit (GRU) BIBREF4 
Similar to the encoder, the recurrent decoder generates one target word INLINEFORM0 to form a translated target sentence INLINEFORM1 in the end. At the time INLINEFORM2 , it takes the previous hidden state of the decoder INLINEFORM3 , the previous embedded word representation INLINEFORM4 and a time-specific context vector INLINEFORM5 as inputs to calculate the current hidden state INLINEFORM6 : INLINEFORM7 
Again, INLINEFORM0 is the recurrent activation function of the decoder and INLINEFORM1 is the shared word embedding matrix of the target sentences. The context vector INLINEFORM2 is calculated based on the annotation vectors from the encoder. Before feeding the annotation vectors into the decoder, an attention mechanism is set up in between, in order to choose which annotation vectors should contribute to the predicting decision of the next target word. Intuitively, a relevance between the previous target word and the annotation vectors can be used to form some attention scenario. There exists several ways to calculate the relevance as shown in BIBREF5 , but what we describe here follows the proposed method of BIBREF2 DISPLAYFORM0 
In BIBREF2 , this attention mechanism, originally called alignment model, has been employed as a simple feedforward network with the first layer is a learnable layer via INLINEFORM0 , INLINEFORM1 and INLINEFORM2 . The relevance scores INLINEFORM3 are then normalized into attention weights INLINEFORM4 and the context vector INLINEFORM5 is calculated as the weighted sum of all annotation vectors INLINEFORM6 . Depending on how much attention the target word at time INLINEFORM7 put on the source states INLINEFORM8 , a soft alignment is learned. By being employed this way, word alignment is not a latent variable but a parametrized function, making the alignment model differentiable. Thus, it could be trained together with the whole architecture using backpropagation.
One of the most severe problems of NMT is handling of the rare words, which are not in the short lists of the vocabularies, i.e. out-of-vocabulary (OOV) words, or do not appear in the training set at all. In BIBREF6 , the rare target words are copied from their aligned source words after the translation. This heuristic works well with OOV words and named entities but unable to translate unseen words. In BIBREF7 , their proposed NMT models have been shown to not only be effective on reducing vocabulary sizes but also have the ability to generate unseen words. This is achieved by segmenting the rare words into subword units and translating them. The state-of-the-art translation systems essentially employ subword NMT BIBREF7 .
One of the most severe problems of NMT is handling of the rare words, which are not in the short lists of the vocabularies, i.e. out-of-vocabulary (OOV) words, or do not appear in the training set at all.
Answer: How to handle the rare words which do not appear in the short lists of the vocabulary.