user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A comprehensive analysis on attention models

Paper content:
Sequence-to-sequence attention-based models are a promising approach for end-to-end speech recognition.
The increased model power makes the training procedure more difficult, and analyzing failure modes of these models becomes harder because of the end-to-end nature.
In this work, we present various analyses to better understand training and model properties.
We investigate on pretraining variants such as growing in depth and width, and their impact on the final performance, which leads to over 8% relative improvement in word error rate.
For a better understanding of how the attention process works, we study the encoder output and the attention energies and weights.
Our experiments were performed on Switchboard, LibriSpeech and Wall Street Journal.
The encoder-decoder framework with attention BID34 BID60 has been successfully applied to automatic speech recognition (ASR) BID26 BID61 BID58 BID47 and is a promising end-to-end approach.
The model outputs are words, sub-words or characters, and training the model can be done from scratch without any prerequisites except the training data in terms of audio features with corresponding transcriptions.In contrast to the conventional hybrid hidden Markov models (HMM) / neural network (NN) approach BID8 Morgan, 1994, Robinson, 1994] , the encoder-decoder model does not model the alignment explicitly.
In the hybrid HMM/NN approach, a latent variable of hidden states is introduced, which model the phone state for any given time position.
Thus by searching for the most probable sequence of hidden states, we get an explicit alignment.
There is no such hidden latent variable in the encoder decoder model.
Instead there is the attention process which can be interpreted as an implicit soft alignment.
As this is only implicit and soft, it is harder to enforce constraints such as monotonicity, i.e. that the attention of future label outputs will focus also only to future time frames.
Also, the interpretation of the attention weights as a soft alignment might not be completely valid, as the encoder itself can shift around and reorder evidence, i.e. the neural network could learn to pass over information in any possible way.
E.g. the encoder could compress all the information of the input into a single frame and the decoder can learn to just attend on this single frame.
We observed this behavior in early stages of the training.
Thus, studying the temporal "alignment" behavior of the attention model becomes more difficult.Other end-to-end models such as connectionist temporal classification BID21 has often been applied to ASR in the past BID20 BID23 BID35 BID1 BID51 BID2 BID26 BID63 BID67 .
Other approaches are e.g. the inverted hidden Markov / segmental encoder-decoder model BID5 , the recurrent transducer BID4 BID41 , or the recurrent neural aligner .
Depending on the interpretation, these can all be seen as variants of the encoder decoder approach.
In some of these models, the attention process is not soft, but a hard decision.
This hard decision can also become a latent variable such that we include several choices in the beam search.
This is also referred to as hard attention.
Examples of directly applying this idea on the usual attention approach are given by BID43 , BID0 , , BID33 , BID27 .We
study recurrent NN (RNN) encoder decoder models in this work, which use long short-term memory (LSTM) units BID24 . Recently
the transformer model BID57 gained attention, which only uses feed-forward and self-attention layers, and the only recurrence is the label feedback in the decoder. As this
does not include any temporal information, some positional encoding is added. This is
not necessary for a RNN model, as it can learn such encoding by itself, which we demonstrate later for our attention encoder.We study attention models in more detail here. We are
interested in when, why and how they fail and do an analysis on the search errors and relative error positions. We study
the implicit alignment behavior via the attention weights and energies. We also
analyze the encoder output representation and find that it contains information about the relative position and that it specially marks frames which should not be attended to, which correspond to silence.2 Related
work BID25 analyzes individual neuron activations of a RNN language model and finds a neuron which becomes sensitive to the position in line. BID7 analyzed
the hidden activations of the DeepSpeech 2 BID1 ] CTC end-to-end system and shows their correlation to a phoneme frame alignment. BID36 analyzed
the encoder state and the attention weights of an attention model and makes similar observations as we do. Attention plots
were used before to understand the behaviour of the model BID15 . BID6 performed
a comparison of the alignment behavior between hybrid HMM/NN models, the inverted HMM and attention models. BID42 investigate
the effects of varying block sizes, attention types, and sub-word units. Understanding the
inner working of a speech recognition system is also subject in , where the authors examine activation distribution and temporal patterns, focussing on the comparison between LSTM and GRU systems.A number of saliency methods BID50 BID32 BID52 are used for interpreting model decisions.
We provided an overview of our recent attention models results on Switchboard, LibriSpeech and WSJ.
We performed an analysis on the beam search errors.
By our improved pretraining scheme, we improved our Switchboard baseline by over 8% relative in WER.
We pointed out the high training variance of attention models compared to hybrid HMM/NN models.
We analyzed the encoder output and identified the representation of the relative input position, both clearly visible in the PCA reduction of the encoder but even represented by individual neurons.
Also we found indications that the encoder marks frames which can be skipped by decoder, which correlate to silence.

assistant: text: improved pretraining, and analysing encoder output and attention