user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Addressing the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts

Paper content:
The transformer is a state-of-the-art neural translation model that uses attention to iteratively refine lexical representations with information drawn from the surrounding context.
Lexical features are fed into the first layer and propagated through a deep network of hidden layers.
We argue that the need to represent and propagate lexical features in each layer limits the model’s capacity for learning and representing other information relevant to the task.
To alleviate this bottleneck, we introduce gated shortcut connections between the embedding layer and each subsequent layer within the encoder and decoder.
This enables the model to access relevant lexical content dynamically, without expending limited resources on storing it within intermediate states.
We show that the proposed modification yields consistent improvements on standard WMT translation tasks and reduces the amount of lexical information passed along the hidden layers.
We furthermore evaluate different ways to integrate lexical connections into the transformer architecture and present ablation experiments exploring the effect of proposed shortcuts on model behavior.
Since it was first proposed, the transformer model BID28 ) has quickly established itself as a popular choice for neural machine translation, where it has been found to deliver state-ofthe-art results on various translation tasks BID3 .
Its success can be attributed to the model's high parallelizability allowing for significantly faster training compared to recurrent neural networks , superior ability to perform lexical disambiguation, and capacity for capturing long-distance dependencies on par with existing alternatives BID26 .Recently
, several studies have investigated the nature of features encoded within individual layers of neural translation models BID1 BID2 . One central
finding reported in this body of work is that, within current architectures, different layers prioritize different information types. As such, lower
layers appear to predominantly perform morphological and syntactic processing, whereas semantic features reach their highest concentration towards the top of the layer stack. One necessary
consequence of this distributed learning is that different types of information encoded within input representations received by the translation model have to be transported to the layers specialized in exploiting them. Within the transformer
encoder and decoder alike, information exchange proceeds in a strictly sequential manner, whereby each layer attends over the output of the immediately preceding layer, complemented by a shallow residual connection. For input features to
be successfully propagated to the uppermost layers, the translation model must therefore store them in its intermediate representations until they can be processed. By retaining lexical
content, the model is unable to leverage its full representational capacity for learning new information from other sources, such as the surrounding sentence context. We refer to this limitation
as the representation bottleneck.To alleviate this bottleneck, we propose extending the standard transformer architecture with lexical shortcuts which connect the embedding layer with each subsequent self-attention sub-layer in both encoder and decoder. The shortcuts are defined as
gated skip connections, allowing the model to access relevant lexical information at any point, instead of propagating it upwards from the embedding layer along the hidden states.We evaluate the resulting model's performance on multiple language pairs and varying corpus sizes, showing a consistent improvement in translation quality over the unmodified transformer baseline. Moreover, we examine the distribution
of lexical information across the hidden layers of the transformer model in its standard configuration and with added shortcut connections. The presented experiments provide quantitative
evidence for the presence of a representation bottleneck in the standard transformer and its reduction following the integration of lexical shortcuts. While our experimental efforts are centered around
the transformer, the proposed components are compatible with other multi-layer NMT architectures.The contributions of our work are therefore as follows:1. We propose the use of lexical shortcuts as a simple
strategy for alleviating the representation bottleneck in neural machine translation models.2. We demonstrate significant improvements in translation
quality across multiple language pairs as a result of equipping the transformer with lexical shortcut connections.3. We report a positive impact of our modification on the
model's ability to perform word sense disambiguation.4. We conduct a series of ablation studies, showing that
shortcuts are best applied to the self-attention mechanism in both encoder and decoder.2 Proposed Method 2.1 Background: The transformerAs defined
in BID28 , the transformer is comprised of two sub-networks, the encoder and the decoder. The encoder coverts the received source language sentence
into a sequence of continuous representations containing translation-relevant features. The decoder, on the other hand, generates the target language
sequence, whereby each translation step is conditioned on the encoder's output as well as the translation prefix produced up to that point. Both encoder and decoder are composed of a series of identical
layers. Each encoder layer contains two sub-layers: A self-attention mechanism
and a position-wise fully connected feed-forward network. Within the decoder, each layer is extended with a third sub-layer responsible
for attending over the encoder's output. In each case, the attention mechanism is implemented as multihead, scaled dot-product
attention, which allows the model to simultaneously consider different context sub-spaces. Additionally, residual connections between neighboring layers are employed to aid with
signal propagation.In order for the dot-product attention mechanism to be effective, its inputs first have to be projected into a common representation sub-space. This is accomplished by multiplying the input arrays H S and H T by one of the three weight
matrices K, V , and Q, as shown in Eqn. 1-3, producing attention keys, values, and queries, respectively. In case of multi-head attention
, each head is assigned its own set of keys, values, and queries
with the associated learned projection weights. DISPLAYFORM0 In case of encoder-to-decoder attention, H T corresponds to the final encoder states
, whereas H S is the context vector generated by the preceding self-attention sub-layer. For self-attention, on the other hand, all three operations are given the output of the preceding
layer as their input. Eqn. 4 defines attention as a function over the projected representations. DISPLAYFORM1 To prevent
the
magnitude of the pre-softmax dot-product from becoming too large, it
is divided by the square root of the total key dimensionality d k . Finally, the translated sequence is obtained by feeding the output of the decoder through a softmax
layer and sampling from the produced distribution over target language tokens.
In this paper, we have proposed a simple yet effective method for widening the representation bottleneck in the transformer by introducing lexical shortcuts.
Our modified models achieve up to 1.4 BLEU (0.9 BLEU on average) improvement on 5 standard WMT datasets, at a small cost in computing time and model size.
Our analysis suggests that lexical connections are useful to both encoder and decoder, and remain effective when included in smaller models.
Moreover, the addition of shortcuts noticeably reduces the similarity of hidden states to the initial embeddings, indicating that dynamic lexical access aids the network in learning novel, diverse information.
We also performed ablation studies comparing different shortcut variants and demonstrated that one effect of lexical shortcuts is an improved WSD capability.The presented findings offer new insights into the nature of information encoded by the transformer layers, supporting the iterative refinement view of feature learning.
In future work, we intend to explore other ways to better our understanding of the refinement process and to help translation models learn more diverse and meaningful internal representations.

assistant: text: Equipping the transformer model with shortcuts to the embedding layer frees up model capacity for learning novel information.