user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Faster and Just As Accurate: A Simple Decomposition for Transformer Models

Paper content:
Large pre-trained Transformers such as BERT have been tremendously effective for many NLP tasks.
However, inference in these large-capacity models is prohibitively slow and expensive
. Transformers are essentially a stack of self-attention layers which encode each input position using the entire input sequence as its context
. However, we find that it may not be necessary to apply this expensive sequence-wide self-attention over at all layers
. Based on this observation, we propose a decomposition to a pre-trained Transformer that allows the lower layers to process segments of the input independently enabling parallelism and caching
. We show that the information loss due to this decomposition can be recovered in the upper layers with auxiliary supervision during fine-tuning
.  We evaluate de-composition with pre-trained BERT models on five different paired-input tasks in question answering, sentence similarity, and natural language inference
.  Results show that decomposition enables faster inference (up to 4x), significant memory reduction (up to 70%) while retaining most (up to 99%) of the original performance
. We will release the code at<anonymized url>.
Inference in large Transformer-based NLP models such as BERT (Devlin et al., 2019) requires prohibitively high-levels of compute, making it expensive to support large volume processing in data centers, and almost infeasible to run on resource constrained mobile devices.
These Transformer models create effective representations using self-attention, a mechanism that allows them to effectively account for wide textual contexts.
However, applying self-attention over the entire input for all layers is computationally expensive.
This raises a natural question: Is self-attention over the entire input necessary in all of the layers?
Previous studies (Tenney et al., 2019; Hao et al., 2019; Clark et al., 2019b) have shown that lower layers tend to capture syntactic phenomena that mostly depend on local contexts and that higher layers capture more semantic phenomena that are relevant to downstream tasks, which depend on longer global contexts.
This suggests that considering only local context in lower layers of Transformer and considering full global context in upper layers can provide speedup at a very small cost in terms of effectiveness.
In this work we focus on paired-input NLP tasks such as reading comprehension, natural language inference and sentence pair similarity.
These tasks provide a natural boundary for the locality of text (e.g., question vs. passage in QA).
Because of this natural decomposition in two segments, we can compute representations for lower layers with only the local segment as the context and compute representations for upper layers with both segments as the context.
This decomposition technique has multiple benefits: It allows for parallel processing of each segment, caching of segments that are available offline, and a significant reduction in runtime memory.
Moreover, since the architecture remains largely same, the original pre-trained weights can be reused in the decomposed model.
To compensate for the differences in the decomposed setting, we augment the fine-tuning loss on the target task with a distillation loss that minimizes the output-level as well as layer-level divergences.
We evaluate the decomposition idea using the BERT model on five different pairwise tasks.
The decomposition achieves substantial speedup (2 to 4.3x) and reduction in memory (51.1% to 76.8%) for only small loss in effectiveness (0.2 to 1.8 points).
Moreover, we find that with decomposition the larger BERT model can even run faster than the original smaller BERT model, while still being more accurate.
Transformers have improved the effectiveness of NLP tools by their ability to incorporate large contexts effectively in multiple layers.
This however imposes a significant complexity cost.
In this work, we showed that modeling such large contexts may not always be necessary and leverage this insight to build a decomposition of the Transformer model that provides substantial improvements in inference speed, memory reduction, while retaining most of the original model's accuracy.
This decomposition model provides a simple yet strong starting point for efficient models as NLP moves towards increasingly larger models handling wider contexts.

assistant: text: Inference in large Transformers is expensive due to the self-attention in multiple layers. We show a simple decomposition technique can yield a faster, low memory-footprint model that is just as accurate of the original models.